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ABSTRACT 

In order to provide specialists and families with 
regular feedback on client change, to enhance the clinical 
decision-making skills of family specialists, and to provide target 
groups with information related to program effectiveness, a 12-week 
plan to improve outcome measures and evaluation protocol in a 
preservation and reunification program was proposed and executed. The 
plan included: (1) the selection of comprehensive, relevant 
instruments; (2) in-service staff training in the use of these 
instruments; (3) the revision of program documents; and (4) the 
development of a volunteer role to accommodate the proper collection 
and dissemination of outcome data. Based on analysis of data from a 
specialist questionnaire and client files, it was concluded that 
changes to program evaluation practices were small but promising. The 
findings suggest that family specialists are more familiar and 
satisfied with program instruments and the outcome data package. The 
consistent collection of clinical and research data continues to 
present a challenge, with competing work demands and the failure of 
staff to view evaluation as a priority undertaking as primary 
obstacles. Future evaluation endeavors must therefore include the 
reallocation of program resources, and program support for — and 
recognition of — the evaluator role. Ten appendices present 
instruments and documents used in the program. Contains 45 
references. (TS) 
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Abstract 

The development of outcome measures for a family preservation and reunification program. 
Coates, Gail E., 1995: Practicum Report, Nova Southeastern University, Master's Program for 
Child Care, Youth Care, and Family Support. Descriptors: Evaluation Evaluation/Problems 
Evaluation/Utilization Formative/Evaluation Summative/Evaluation Evaluative/Thinking 
Evaluation/Methods Critical/Thinking Family/Problems Family/Counselling 
Family/Programs Family/Support. 

The problems associated with outcome evaluation practices in a preservation and 
reunification program are similar to those identified in the family support literature and by field 
practitioners. In part, the use of narrow or insensitive outcome measures, inadequate training in 
the use of instruments, and poor staff and consumer investment in the evaluation process have 
interfered with the consistent collection and holistic presentation of meaningful outcome data by 
family specialists alike. 

In order to provide specialists and families with regular feedback on client change, to 
enhance the clinical decision-making skills of specialists, and to provide target groups with 
information related to program effectiveness, a 1 2-week plan to improve outcome measures and 
evaluation protocol in the program was proposed and executed. This included the selection of 
comprehensive, relevant instruments, in-service staff training in the use of the instruments, the 
revision of program documents, and the development of a volunteer role to accommodate the 
proper collection and dissemination of outcome data. These changes are represented in the 
appendices. 

Overall, it was concluded that changes to program evaluation practices were small but 
promising. The findings suggest that specialists are more familiar and satisfied with program 
instruments and the outcome data package. A few reported using the data to enhance case 
planning and clinical decision-making. Further, management has adopted these practices as an 
integral part of program technology. Conversely, the consistent collection of clinical and 
research data, particularly at case follow-up, continues to present a challenge. Competing work 
demands and the failure of staff to view evaluation as a priority undertaking are primary 
obstacles. Future evaluation endeavors must therefore include the reallocation of program 
resources, and program support for and recognition of the evaluator role. 
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CHAPTER 1 
INTRODUCTION 

This chapter provides an overview of the problem setting and my corresponding role. 

The Setting 

The practicum project focused on a family support program operating under the auspices of 
a local child and family service agency. The nonprofit private agency experienced its beginnings 
over 30 years ago as a residential treatment and education facility for children experiencing a 
variety of physical, behavioral, and emotional problems. Since then, it has spawned a host of 
programs, including treatment foster care, semi-independent living, and receiving and teaching 
homes, in response to the growing need for a continuum of community services. The most 
recent addition is a 7-year-old home-based family support program for high-risk children and 
families. Based on a philosophy that all families require support and that whenever possible 
family members belong together, their goal is to help preserve the family unit; to successfully 
reunite children and families; and as a last resort, to strengthen and maintain family relationships 
when out-of-home placement is necessary. Program statistics reveal that 85% of the referred 
children contacted at 1-vear follow up are residing with immediate family, relatives, or friends. 
The support program offers a range of services to families in the rural and urban Calgary region, 
including a native program for North American Indians; a long-term program for families with 
chronic problems and complex needs; and a short-term program, which accommodates the 
largest staff and consumer group and serves families with a number of presenting problems. It is 
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this short term program that served as the clinical setting for the pioject. 

The program is contracted by Alberta Family and Social Services to provide short term, 
intensive in-home support and 24-hour crisis support to a mini turn of 650 children and their 
families who are referred annually by the Child Welfare Division. Each specialist carries a 
two-family caseload; and each family receives approximately 20 hours of weekly support in their 
home environment over a period of 8-12 weeks, with follow-up support available as needed. 

The referred children range in age from birth to 1 8 years. 

Two of the agency's managing directors oversee the operation of the short term program, the 
native unit, and the long term program. These programs operate within a three-tiered 
organizational structure that consists of program directors responsible for administrative 
functions; coordinators who provide ongoing training, supervision, and consultation to front line 
staff; a volunteer coordinator in charge of recruiting and training program volunteers; family 
specialists, who provide direct support and treatment to children and families; program 
assistants, who aid specialists in carrying out treatment as required, and a full-time 
trainer-evaluator who was recently recruited to assess and revise program practices and 
procedures. Until 2 years ago, organizational structure, staffing patterns, and resources reflected 
contractors' beliefs that only extremely high-risk cases (the top 1% of "needy" families) 
warranted Ireatment. Since then, staff size has increased nearly threefold at the front-line level 
and proportionately at the management level; this is in response to an increase in funding dollars 
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for family support services and an awareness of the need to offer cost-effective, intensive, 
preventative services to families who might otherwise join the ranks of the higher-risk group. 
There are currently 1 1 family specialists serving the short term program. 

An analysis of client profiles reveals that program families come from a range of 
socioeconomic, cultural, and educational backgrounds; present differently; and require support 
services that vary in degree and kind. In addition, they demonstrate various levels of 
engagement, cooperation, trust in professionals, and personal experience with formal systems. 
The heterogeneous nature of families necessitates a respect for family values and cultures, the 
active participation of families in their own treatment, and the individualization of services. 
Program emphasis is on creating and building on family members' strengths, and causal 
inferences and labeling are deliberately avoided. 

The Family Specialist Role 

Although specialists typically hold a bachelor's degree in the social sciences, each works as 
a member of a multi-disciplinary team that offers a range of educational, intra-agency, and field 
experience to the program. All specialists receive extensive pre-service training and ongoing 
training in the Teaching Family Model (TFM) of service delivery, acquire ongoing practical 
experience, receive regular consultation advice, and are encouraged by the agency to further 
develop professional competence through participation in program workshops, program 
development opportunities, continuing education courses, and inter- and intra-agency seminars. 
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The TFM, developed in 1967 and used widely by more than 30 Teaching Family Association 
agencies and programs and various non-members reflects a behavioral model of intervention, 
although a variety of cognitive and systems frameworks are incorporated into practice. 

Regarding role expectations, trained specialists collaborate with families, assigned child 
welfare workers, community professionals, and program staff to maximize effective case 
management, alleviate child welfare concerns, and improve family functioning. The first 2 
weeks of the intervention focus on relationship development, family assessment, and the 
formulation of a comprehensive family treatment plan that is used to guide the treatment 
process. 

The primary role of the specialist involves skill teaching. Specifically, specialists work 
with participants on areas such as parenting, child behavior, rational decision-making, 
communication, and basic needs; and they facilitate the family's demonstration, maintenance and 
generalization of these skills. In addition, families leam new ways of viewing themselves and 
others, and of impacting their environment. This process is meant to empower, mobilize, and 
strengthen families to deal with ongoing anticipated and unanticipated life events that may 
otherwise prove stressful and detrimental. Other roles include activating temporary and 
long-term resources; serving as a case-level advocate; basing with involved professionals and 
significant others; educating parents on child development and parental responsibilities; 
providing concrete assistance; and providing emotional support and counselling. An effective 
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specialist will create opportunities for learning and take advantage of existing situations, 
including crises to help the family acquire new skills. The specialist is available 24 hours daily 
to respond to crises, operates on a flexible work schedule, and offers follow-up support to former 
families as required. 

In terms of administrative duties, documentation is considered an important specialist 
accountability and program measure. At case opening information on family history, 
demographic variables, and problematic behaviors are recorded on an intake form. In addition, 
program parents are required to sign consent forms to partake in counselling and for the release 
of family information; and are encouraged to consent to the transport of children, the possible 
use of physical restraints on children, and participation in family-based research. In order to 
participate in research, parents must complete Achenbach's Child Behavior Checklist and page 1 
of the Parent-Child Behavior Questionnaire at intake, at case closure, and occasionally at 
4-month follow-up intervals. This data is compiled by the program statistician and is used as 
program feedback. 

In addition, daily contact notes and time sheets provide a record of treatment activities and 
treatment intensity. At case closure, a discharge form describing child placement and family 
"prognosis", and a termination letter documenting the progress toward goals are presented to 
child welfare. Over a 12-month follow-up period, families are contacted bimonthly and asked 
to provide child placement information. Occasionally, former participants are contacted by 
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available staff and asked to qualitatively describe which of the skills acquired during treatment 
are proving helpful, relevant, and replicable. Statistics on child placement at case closure and 
1-year follow-up are shared with contract managers; the remaining data is compiled by the 
program statistician and used for practitioner and program feedback. 

Due to developmental growth within the family support field and the short term program, 
the program is currently undergoing significant changes to its policies and procedures in areas 
such as formal staff training and evaluation, working conditions, and treatment documentation. 
By encouraging specialists' involvement in these processes, the program has recently witnessed 
improvements to staff practices, competence, feelings of empowerment, credibility, and morale. 
Having been certified according to Teaching Family Association (TFA) standards for my 
demonstration of competence in the area of family treatment, I am an active participant on the 
agency’s Evaluation Review Committee, on the program's staff evaluation committee, and in the 
formal practitioner group meetings; and I serve as the Practitioner Representative for the 
home-based division of TFA, and as a representative on TFA's Board of Directors. Given the 
benefits of staff involvement, it is not surprising that the proposal to develop evaluative outcome 
measures in the short term program was widely approved following a discussion with 
upper-management bodies. Program staff were to be kept abreast of project developments. 
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CHAPTER 2 
THE PROBLEM 

This chapter describes the problems associated with outcome evaluation practices in the 
short term program, and discusses their relevance to other field practices. Information was 
obtained from literature sources and from practitioner colleagues and coworkers. 

The Problems Associated With Program Evaluation Practices 

Evaluation practices within the short term program were inadequate. Specifically, the 
outcome measures utilized by program staff were inappropriate, their application was 
inconsistent, and the outcome results were not properly utilized. 

The program used page 1 of the Parent-Child Behavior Questionnaire (PCBQ) and 
Achenbach's Child Behavior Checklist (CBCL) (see Appendix A); open-ended parent 
questionnaires on family skills; and closed-ended questions on child placement, to assess child 
and family outcome variables. To some degree, each of these instruments played a role in the 
inadequate program evaluation practices described above: First, the inappropriateness of the 
instruments included the use of the CBCL. This measure, which is known for its ability to 
distinguish between clinical and nonclinical populations and among various types of clinical 
populations, does not effectively measure changes within clinical families like the high-risk 
families commonly enrolled in the short term program. Second, in reference to the inappropriate 
application of program instruments, there was an absence of consistent, timely, and regular 
interval testing involving all of the above measures: The PCBQ and CBCL were not issued to all 
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program families; further, specialists did not regularly implement the measures at case opening, 
case closure, and 4-month follow-up periods. In addition, family skill and child placement data 
were not consistently collected at regular intervals over 12 months of follow up with all program 
families. Finally, with respect to the inappropriate utilization of outcome data, the PCBQ and 
CBCL served a limited research function, so that none of the information was used as feedback 
for clinical decision-making by specialists; the data obtained from the open-ended questionnaires 
on skill demonstration were frequently not communicated to specialists; and global child 
placement data that did not include other relevant outcomes were used in the absence of other 
outcome data to justify program effectiveness to a variety of target groups, including specialists, 
professionals, and contract managers. It is important to note that while program staff collect 
specific data on child placement, the favorability of child placement was not ascertained. Of the 
three kinds of problems discussed here, the last two are the most significant, since the 
development of adequate instruments would have no positive affect on evaluation practices 
unless the problems associated with data collection and utilization were adequately dealt with. 

The program needed to utilize evaluative outcome measures; the outcome measures needed 
to be used at regular intervals during case opening, closure, and follow-up with all program 
families; and the results needed to be shared collectively with target groups, and to contribute to 
research and serve as clinical feedback during treatment interventions. 
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Literature Findings 

Evaluation constitutes a multi-disciplinary, diverse, complex, and controversial practice 
(Scriven, 1993) that has been described as "...a process of delineating, obtaining, and providing 
useful information forjudging decision alternatives" (Dunst, 1991, p. 15). Several authors have 
attempted to bridge the gap between evaluation practices in order to emphasize their similarities 
and compatibility: For example, following an analysis of three heterogeneous studies, Yin 
(1994) concluded that effective evaluation studies represent a "singular craft" that is 
comprehensive and in-depth, successfully undermines the credibility of competing hypotheses, 
allows one to derive broad implicaions from evaluation results, and reflects a competent 
analysis of available data. Other authors (Powell, 1987; Reichardt & Rallis, 1994; and Royse, 
1990) have argued that qualitative and quantitative study measures are complementary and 
contribute to the richness and accuracy (validity and reliability) of data collection and 
interpretation. Further, resistance and barriers to using rigorous experimental designs in natural 
settings (Powell, 1987) have lead to general consensus around the use of quasi-experimental 
designs to improve the reliability and validity of various methodologies and results, and to 
accommodate ethical considerations in research and program evaluation. 

Despite these attempts, the social science field has been among the most criticized by 
traditionalists (Scriven, 1993) for its use of evaluation; critics claim that evaluation is marred by 
evaluator biases and political influences that can lead to subjectivity and inference. Scriven 
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(1993) contends that in reality, many social science studies are grounded in fact and logic, and 
that the inability of evaluators to adequately rationalize their conclusions is one problem all 
disciplines must confront. Nevertheless, this criticism is not lost on the field: To minimize the 
problem of evaluator and program bias, some programs (Jacobs, 1988) have engaged the services 
of outside evaluators to assess program effectiveness. Other individuals have attempted to 
minimize the problem of "scripted thinking" and worker bias: For example, in a study by 
Waxman, Rapagna, and Dumont (1991), the authors offer suggestions around educating 
counsellor-trainees to perform objective client assessments in ways that accurately depict the 
complexity of the human condition. The desirable evaluative attributes described earlier by Yin 
(1994) are consistent with these approaches. 

The growing popularity of evaluation studies in the social science fields is evident from the 
numerous "program, product, personnel, policy, performance, and proposal" (Scriven, 1993, 
p. 310) evaluations being performed. For example, countless studies have been executed by 
education researchers to deal with matters such as bilingualism and the development of 
educational practices and theories (Fetterman, 1986). While program managers are obviously 
concerned with their own agenda, Scriven (1993) advocates that the needs of program 
participants be considered first and foremost. In this context, evaluation is an essential means of 
assessing the value or impact of a particular product or service on its recipients. This is 
especially relevant for the human service and helping professions. 
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Two kinds of evaluations are commonplace in family support programs and organizations: 
Respectively, formative and summative evaluations, which enhance program development and 
determine outcome effects on consumers and their larger systems (Littell, 1986) are compatible 
and have numerous advantages. Together, they support logical assumptions and generate factual 
information that can lead to quality assurance; increase program and staff accountability, 
credibility, and visibility; contribute to field practices and theories; enhance satisfaction and 
cooperation among funders and policy makers; facilitate program planning and replication; and 
most importantly, offer appropriate responses to consumer needs. Out of the growing need to 
secure and maintain funding, to provide effective services to a challenging and complex 
consumer group, and to take advantage of available information on family support and education 
programs, family support researchers and program staff are increasingly making use of these 
evaluation procedures. Many programs, such as the Child Survival/Fair Start for Children 
program for young mothers and pregnant teen-agers (Bond & Halpem, 1988) have conducted 
process evaluations to determine operational effectiveness; and others, such as the Advance 
Parent-Child Program (Rodriguez & Cortez, 1988) have implemented pre- and post-test 
measures to determine the impact of services such as parenting classes on disadvantaged 
families. 

Ideally, effective outcome measures determine "...the extent to which exposure to a 
program changes the "targets" of the program in desired directions. . . significantly beyond .. .those 
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directions that would occur in the absence of the program" (Rossi, 1992, p. 90). Functionally, 
these measures provide education and feedback to programs, workers, and participants during 
the assessment, intervention, and follow-up stages of treatment. In reality however, the 
problems associated with outcome evaluations in family preservation and reunification programs 
are well documented: For example, from his analysis of several family preservation programs, 
Rossi (1992) concluded that to date, programs have not been able to significantly or consistently 
demonstrate their impact on program participants. One major problem is that there is an 
over-reliance on narrow, single outcome measures, and these measures often reflect the 
particular "...political and social context of a program" (Powell, 1987, p. 321). Indeed, programs 
may succumb to social and beaurocratic pressures to alleviate vast social problems or serve as a 
cost-effective alternative to other social services. In addition, the complexity and diversity of the 
field’s programs and families lead to difficulties in developing adequate instruments. This is 
evidenced by the popular use of child placement as a primary outcome measure. While keeping 
families together is a primary goal of preservation and reunification programs, the failure of a 
program to develop broader outcome measures and to appropriately justify their use can distort, 
limit, or mask the positive short term and long term effects of treatment. For example. Head 
Start (Zigler & Freedman, 1987), an early childhood intervention project initially relied on 
changes in child IQ as the primary outcome variable; however, when these temporary and 
unsubstantial effects diminished within a few years of school attendance, the programs were 
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widely criticized. Subsequent evaluation studies have revealed a number of positive outcomes 
for Head Start children, including enhanced social and emotional development. Service delivery 
and outcome measures have since expanded in order to create and capture more complex 
changes to children and families. Notwithstanding these improvements, the family support field 
has yet to develop a repertoire of reliable and valid outcome measures. 

A second problem with outcome studies concerns the way these evaluations are performed 
and operationalized (Cole & Duva, 1990): The timing of outcome evaluations is paramount to 
success (Weiss & Jacobs, 1988); however, many programs conduct evaluations before their 
respective goals and procedures are firmly established. In addition, some programs do not 
conduct follow-up assessments with former clients, which can lead to the misinterpretation of 
outcome effects. One obstacle to follow-up is that families are mobile and difficult to track 
(Kinney, Haapala, & Booth, 1991). Even when assessments are conducted during and following 
an intervention, they are not always performed at regular or frequent intervals. The time and 
energy required by workers to perform unnecessary daily treatment documentation, and the level 
of participant cooperation may exacerbate genuine efforts. Further, the complexity of some 
instruments and their inability to capture subtle client changes (Wasik, Bryant, & Lyons, 1990) 
presents a challenge. This is in spite of competing rationales that regular interval testing can 
increase investment among workers and families, can be used to track gradual positive and 
negative changes in families, and can more accurately depict family outcomes (Wasik, Bryant, & 
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Lyons, 1990). Similarly, as with developing fields such as bilingual education, the absence of 
"...sound and practical guidelines and materials...." (Lam, 1992, p. 18 ) is a common problem in 
the family support field. While the field has begun to develop a handful of valid and reliable 
outcome measures, inconsistencies with respect to their use, coupled with disagreement on the 
targets of assessment (Powell, 1987) and over whether observational or self-report measures 
should be used can distort and limit outcome results. Further, every instrument has its 
advantages and disadvantages, which makes selection a double-edged sword. For example, 
depending on program needs, one may have to balance instrument reliability and sophistication 
with the degree to which it can be easily administered and scored. 

Regarding operational inconsistencies, studies use different definitions of successful child 
placement (such as home versus out of home; with kin versus not with kin), organize their data 
differently (for example, according to number of families with children at home versus the 
number of children at home), and do not distinguish between higher and lower risk groups. In 
addition, the avoidance of out-of-home placement may be directly attributed to the program, 
even when it reflects the unwillingness of parents to place children, the inaccessibility of 
placement services, or the passing of family crises. Further, positive outcomes for family 
members can occur in spite of or because of child placement, and should therefore be 
considered. These methodological and conceptual problems (Cole & Duva, 1990) can lead to 
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Lastly, information gleaned from research studies of family preservation programs is not 
widely disseminated (Cole & Duva, 1990); and inconsistencies with and disagreement over the 
use of specific research designs make it difficult to cross-compare results or to select desirable 
evaluation approaches. In addition, the literature frequently reports narrow child placement 
data. As a result, many programs struggle in isolation and confusion to develop or reinvent 
adequate evaluation measures. Nevertheless, more sophisticated evaluation efforts are currently 
being conducted and documented; For example, the Family Connection Project family support 
program (McCroskey & Nelson, 1989) provides detailed information on their creation and pilot 
of the Family Assessment Form, and encourages other programs to analyze or utilize this 
instrument and provide them with feedback regarding its relevance and utility; and the 
Homebuilders program (Kinney, Haapala, & Booth, 1991 ) has published a book to provide 
others with information on the treatment and support^ffn^-risk families. It offers a description 
of various program outcome studies, in addition to program outcome measures. Information on 
evaluation practices can be useful, as long as methodological approaches and concepts are 
carefully described and defined. 

Information from Practitioners 

To establish the relevance of the literature to the practicum problem and to obtain additional 
information on local family support evaluation practices, a Specialist Questionnaire (see 
Appendix B) was completed by 9 of 10 available specialists in the short term program, and 
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informal telephone interviews were conducted with family specialists from two local agencies. 
Due to the fact that no single Calgary service approximates the intensity of the short term 
program, 1 selected two medium-intensity programs that resemble the short term program along 
several dimensions (such as home visiting). Taken together, several themes emerged from the 
data. 

One common theme involves the adequacy of program outcome measures. A worker from 
one local program stated that until recently, their primary outcome measure was a standardized 
social support measure that consists of the dimensions "function of support 1 and "aspect of 
support". She explained that the instrument is plagued by confusing terminology, making it 
difficult for families to comprehend and accurately respond to the questions, and difficult for 
staff to clarify the questions and interpret the results. Several workshops designed to train staff 
in the use of the instrument failed to alleviate the problems, which led them to conclude these 
measures were inappropriate for the program. As a result, staff were in the process of replacing 
the instrument with two standardized measures that are user-friendly and that depict several 
aspects of social support. The second worker explained that staff members actively participated 
in the creation of a multidimensional instrument that relies on specialist observations and 
includes a key for easy scoring and interpretation. Both workers felt that the usefulness of the 
instruments and staff participation in their development contributed to high levels of staff 
investment, despite the fact the instruments require considerable time to complete. 
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Similarly, information gathered from the Specialist Questionnaire revealed that most 
practitioners in the short term program did not find the PCBQ and the CBCL to be useful 
instruments because they fail to measure important family changes, and they do not provide staff 
with immediate feedback on family progress or on the effectiveness of specific intervention 
strategies. One inadequacy related to the restricted use of data for program development 
research. A second reason concerned the nature of the CBCL: While this child-focused measure 
of social competence and behavioral difficulties is considered comprehensive, valid, and reliable 
(McMahon, 1984), it does not address the impact of systems such as the family on child 
development (Mooney, 1984). Further, because the instrument is sensitive to differences among 
clinical populations, it is primarily used as a screening and outcome evaluation device, rather 
than as a treatment guide (McMahon, 1984). Similar to the short term program. Homebuilders 
(Kinney, Haapala, & Booth, 1991) utilizes the CBCL as part of its evaluation package; however, 
the weaknesses of this measure are balanced by the use of clinical measures that can provide 
staff and families with regular, important feedback. While these measures are simple to 
administer, most of the specialists in the short term program reported that they are capable of 
performing clinical assessments but have not been adequately trained in the administration and 
scoring of these instruments; this can also contribute to staff perceptions that the measures are 
inadequate. With respect to child placement measures, specialists concurred that placement by 
itself is an insufficient outcome measure; and the majority of them felt pressured to ensure the 
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in-home placement of a child even if it was not in the family's best interest. Collectively, 
information provided in the literature and by family specialists indicated a need for the selection 
of appropriate outcome measures. For the short term program, clinically-relevant, practical, 
comprehensive outcome measures were required. 

Another theme that emerged was the need for improved evaluation protocol, including the 
consistent and frequent use of instruments by family specialists. Regarding consistency, one 
worker reported that of three outcome measures utilized by program staff, no single measure or 
combination of measures was being consistently utilized by specialists; and decisions for their 
use were presumably being based on the specialist's judgment, rather than on specific program 
guidelines. Further, no follow-up data was being collected on families, because upper 
management believes that former-client contacts are intrusive reminders to families that 
problems once existed. The second worker reported that their multidimensional observational 
measure was being issued at case opening and closure; however, 4- and 6-month fohow-up 
families were simply being asked to update specialists on their situation in narrative form, which 
affects the consistency of pre- and post-data collected. He also stated that case families are not 
aware that they are participating in observational assessments; while this can decrease family 
pressure to "rate" positively, it can restrict family investment in the treatment process, and it 
raises ethical concerns. Both workers rationalized that family mobility, worker time restraints, 
and limitations imposed by telephone contact interfere with the consistent, .cgular use of 
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outcome measures. 

Specialists in the short term program also revealed inconsistencies with respect to the 
implementation of outcome measures: Some of the specialists reported their failure to request 
client participation in program research endeavors; and the majority of specialists admitted that 
they did not regularly implement research-based outcome measures with consenting families at 
case opening and case closure, and that they never did so at 4-month follow-up periods. Also, 
they and other program staff seldom collected child placement data during follow-up intervals. 
Staff indicated that it is difficult to implement the measures given t ieir work demands, and that 
they do not feel the benefits of research are worthy of their efforts. Lack of knowledge regarding 
their use may also have been a contributing factor. Regardless of the number of individual-case 
families that agree to participate in research, most of the specialists attributed low levels of 
family investment to lack of understanding of the process, lack of trust, and the absence of direct 
benefits to their families. Given these criticisms, the consistent and frequent use of outcome 
measures was expected to improve as evaluation protocol and measures that provide practical 
feedback were introduced. 

A related theme emerging from the specialist information was the appropriate organization 
and use of program outcome data. Programs prioritize and use outcome data in different ways. 
For example, in the medium intensity r r ograms the primary outcome variable is the extent to 
which participants mobilize and utilize community supports. In the short term program the 
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primary goal is to preserve the family unit, which dictates the collection and dissemination of 
child placement data. Unfortunately, outside pressures such as limited funding can affect the 
way programs define success. Specialists in the short term program indicated that they have 
never included outcome data in their professional documents, shared outcome data with others 
during case conferences, or received child placement feedback from program staff. Most 
criticized the outcome data package (basic child placement data) for its inability to provide 
complete family information to target groups. Data needed to be clearly described and 
collectively presented, in order to improve evaluation protocol. 

The descriptions of problems specific to this program were consistent with what was 
described in the literature and by local professionals. While the programs mentioned above had 
resolved some of the problems associated with evaluation practices, the short term program 
needed to address issues such as the usefulness of the evaluation package, staff training in the 
use of program outcome measures, guidelines for evaluation protocol, and the dissemination of 
relevant information to target groups Therefore, in Chapter 4, information gleaned from a 
literature review was used to guide the development of evaluative outcome measures and 
practices in the short term program, including solutions to evaluation problems, the selection of 
promising measures and approaches, and rationales for their implementation. 




26 



CHAPTER 3 

GOALS AND OBJECTIVES 

This chapter outlines the goals and objectives that served as the basis for this practicum 
project. 

Practicum Goal 

The aim was to develop and implement appropriate evaluative outcome measures for use in 
the short term program. The literature review and professional testimonies described in Chapter 
2 provided valuable insight into the problems commonly associated with outcome measures used 
in family support programs. Of these difficulties, the use of measures that are too narrow in 
scope or that fail to depict important changes in program participants can significantly affect the 
adequacy of outcome measures. Program priorities, the complexity of family work, and the 
limited availability of evaluative instruments in part provide a context for the existence of these 
problems. Given the importance of meeting participant needs, there is little question that 
comprehensive evaluation measures are a necessary and desirable alternative. 

The aforementioned information sources also revealed that the methodological problems 
commonly associated with program outcome evaluations in part stem from the complex nature 
of the programs, the families, and the field. These problems include the absence of standard 
guidelines and models, the inconsistent use of pre- and post-test measures, and the restricted use 
of outcome data. Limited staff and family investment in the evaluation process can exacerbate 
these problems. It was believed that the consistent implementation of outcome measures and the 
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appropriate use of outcome data would alleviate some of the methodological problems 
associated with evaluation practices. 

Objectives 

Based on the above goal, the objectives included the following: 

( 1 ) The program would select and/or develop outcome measures of greater clinical significance 
to practitioners and high-risk populations. These instruments would provide staff with adequate, 
relevant feedback on family changes and progress toward goals, as measured by the specialists' 
responses to a questionnaire 1 designed. 

(2) Program specialists would become more familiar with how to implement program measures 
and how to interpret the outcome results, as evidenced by specialists' responses to the Specialist 
Questionnaire 

(3) The use of program measures by specialists during client intervention and follow-up 
phases would increase in the following ways: 

(a) The percentage of specialists utilizing the measures would increase. 

(b) Specialists would utilize the measures at more consistent intervals with case families. 

(c) Specialists would utilize the measures at more frequent intervals with case families. 
These objectives would be measured using data gathered from Specialist Questionnaire and 



client files. 
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CHAPTER 4 
SOLUTIONS 

This chapter focuses on solutions to the problem of inadequate evaluation practices in the 

short term program. Included is a review of professional sources that facilitated the 
identification of evaluative outcome measures and approaches. A proposed plan for their 

implementation is then described, and systematically presented in calendar form. 

Criteria for Instrument Selection 

Family assessment measures can be used to determine the severity of family difficulties, to 
provide workers with consistent information on family change and service needs, and to enhance 
workers' clinical decision-making (Gabor, Thomlison, & Hudson, 1993). In Chapter 2, the Child 
Behavior Checklist was criticized for its inability to capture important changes taking place 
within the program's high-risk families. In order to identify more appropriate clinical outcome 
measures for use in the short term program, it was helpful to consider some of the decisions 
faced by other evaluators and family support staff during the selection process. 

First, Walker and Crocker (1988) organize family assessment measures into four categories: 

face-to-face interviews, projective techniques, observations, and self-report scales. Of these, 
self- report measures are among the most widely used. The reasons for their popularity exceed 
the relative ease and inexpense with which they are administered: Many investigators strongly 
believe that clients should be active participants in their own assessment and treatment. For 
example, Kinney, Haapala, and Booth explain that "the families have the best data about their 
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situation" ( 1 99 1 , p. 8 1 ); and Corcoran & Fischer (1987) refer to the client as a "very good 
observer" who can provide others with less available types of information Also, Dunst, 
Trivette, and Deal's (1988) helping model of family assessment and intervention accommodates 
the use of self-report practices to facilitate "supportive encounters". They argue that outside 
evaluators who impose their own views on families run the risk of creating "oppositional 
encounters" that can impede family progress. Other authors (for example, Gabor, Thomlison, & 
Hudson, 1993) emphasize the importance of trusting worker-client relationships to facilitate 
honest disclosures from clients, and Rankin (1990) views deliberate and unintentional "client 
dissimulation" as an opportunity to promote client openness and change. 

Other authors (for example, Weiss & Jacobs, 1988) hold that the use of observational 
techniques by trained outside evaluators is a more effective means of objectively measuring 
complex and subtle family interactions, despite the fact that these approaches are costly and 
occasionally spark opposition by program staff (Ellwood, 1988). They reason that self 
reports are susceptible to bias, and that evaluators disagree on whether reports should be 
aggregated or examined independently (Sawyer, Sarris, Baghurst, Cross, & Kalucy, 1988). 
Specifically, some authors believe that family members (Besharov & Baehler, 1992) and 
practitioners (Cole & Duva, 1990) are incapable of adequately assessing family difficulties, 
strengths, and needs, even though family members may alter their behavior during formal 
observations (Achenbach, 1985), and trained observers will classify or interpret behaviors in 
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different ways, which poses a similar threat to content validity (Weiss & Frohman, 1985). 

To improve the accuracy of the information collected, Cole and Duva ( 1990) recommend 
that significant others, in addition to family members and workers conduct assessments of 
client needs and change. The selection of individuals for this comprehensive approach will 
depend on their capabilities (Corcoran & Fischer, 1987) and availability, as well as the 
availability of funding (Cole & Duva, 1990). 

A second decision confronting evaluators involves the selection of existing instruments or 
the creation of new instruments. Corcoran and Fischer (1987) present their readers with criteria 
for instrument selection, and list numerous standardized child, adult, and family measures. 

The advantages to utilizing established measures include known reliability and validity, 
effective management of program resources (such as time and money), and the ability to conduct 
cross-program and participant studies. In cases where suitable standardized measures are not 
available, programs such as the Calgary Integrated Services and the Family Connection Project 
(McCroskey & Nelson, 1989) have elected to develop their own measures. Powell ( 1987) 
recommends that new measures be piloted in conjunction with more established measures using 
small client samples. Staff may also elect to modify existing measures; however, this will alter 
their original reliability and validity values. 

A third issue is the selection of practical instruments that match program resources. For 
example, because the majority of home-based family support practitioners hold a bachelor's 
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degree or less (Wasik, Bryant, & Lyons, 1990), it would be unrealistic to implement measures 
that require high levels of psychometric or statistical expertise. Similarly, many workers are 
operating under time and energy' restraints. The ease with which rapid assessment instruments 
(RAI’s) are administered and the level of comfort with which clients can express themselves 
make these measures popular; however, Corcoran and Fischer (1987) caution that the misuse of 
RAl’s (or any other instrument) can affect validity and reliability; can result in an over reliance 
on the measures for decision making; and can create program resistance. Therefore, educating 
staff in the proper use of program measures is paramount. 

Fourth, while the diversity of family needs and services make the selection of relevant, 
comprehensive measures difficult (Powell, 1987), program staff must select measures that reflect 
treatment objectives, and are of theoretical and clinical significance. Due to the nature of family 
preservation, the most common variables studied and measured are classified under family • 
functioning, child risk and behavior, and social support (Cole & Duva, 1990). In addition, 
McFall (1986) points out that evaluators should use theoretical models to guide measurement 
selection, and that outdated or inappropriate instruments should be replaced with practice 
models that contribute to and accommodate new information. Another option presented by 
Corcoran and Fischer ( 1 987) is the selection of measures that are not derived from any specific 
theoretical orientation. For example, Gabor, Thomlison, and Hudson's (1993) Family 
Assessment Screening Inventory measures the presence and severity of family difficulties 
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without providing reasons for their existence or means of alleviating them; users are therefore 
free to operate from within their own theoretical frameworks. 

Finally, as evaluation practices become more complex evaluators increasingly rely on 
multidimensional or several single-dimensional instruments to adequately measure outcome 
variables. Cole and Duva ( 1990) rationalize that appropriate measures can then balance out less 
effective measures. Corcoran and Fischer (1987) recommend the use of measures where the 
target populations resemble program clients. 

Having identified these qualities, it was possible to determine the needs of the short term 
program and to select adequate outcome measures based on these descriptions. Given the 
limited resources (time, money, and technical skill) of the program and specialists, the need 
for comprehensive clinical measures, and the time limit of the project, program staff needed to 
utilize one or more standardized self-report measures that are sensitive to changes within 
program families; are cheaply and easily administered, scored, and interpreted by program 
staff; and depict several relevant dimensions of family life. Several existing self-report measures 
met the above criteria and warrant analysis here. 

Proposed Program Measures 

In order to facilitate the selection of appropriate outcome measures for use in the short term 
program, numerous existing instruments that measure various aspects of family functioning, 
social support, and basic child and family skills were examined. Several promising measures are 
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included for discussion. 

First, improving family functioning is an important goal of the short term program. Two 
suitable instruments, the Family Adaptability and Cohesion Evaluation Scales (FACES II and 
FACES 111) and the McMaster Family Assessment Device (FAD), are described. 

The Circumplex Model developed by Olson (Olson & Portner, 1983) is renowned for its 
widespread use in clinical and research circles. The adaptability and cohesion dimensions are 
represented in curvilinear form, with continuum extremes indicative of poor functioning. 
Communication is represented as a separate index that influences the direction of the two 
dimensions. The corresponding self-report measure FACES II is easy to use, and can be used by 
several family members and professionals to assess current and desired levels of family 
functioning. In addition, the measure can be used in various therapeutic settings, to enhance 
understanding of one's family of origin, and to assess functioning in the context of different 
developmental life stages (Olson & Portner, 1983). Showing fair to modest validity and 
reliability scores (Corcoran & Fischer, 1987), this measure has been criticized in part because of 
the use of small and non-representative samples during research undertakings. In one study 
involving 2,440 families, the authors (Green, Harris, Forte, & Robinson, 1991 ) found "lack of 
support" for the curvilinear relationship as measured by FACES III. These findings are 
supported by Perosa and Perosa (1990). 

The McMaster Model is comprised of seven interdependent components of family 
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functioning that are measured separately using Epstein, Baldwin, and Bishop's (1983) Family 
Assessment Device (FAD) (see Apperdix C). These dimensions include problem solving, 
communication, roles, affective responsiveness, affective involvement, behavior control, and 
general functioning. One study (Miller, Bishop, Epstein, & Keitner, 1985) revealed that the 
measure is valid and reliable, relates well to social desirability, distinguishes clinical from 
nonclinical families, and adequately correlates with FACES II and the Family Unit Inventory. 
Other strengths include its ability to measure actual and desired family functioning levels, its 
utility as a clinical and research instrument, and its user-friendly qualities. Other authors also 
find the measure to be valid and sensitive to differences between clinical and nonclinical 
populations, in addition to response differences between parents and adolescents (Sawyer, Sarris, 
Baghurst, Cross, & Kalucy, 1988); and to be superior in sensitivity to FACES II (Fristad, 1989). 

In light of the recent criticisms, research findings, and the content of these self-report 
measures, the FAD was deemed more appropriate for use in the short term program. It was 
hoped that the strong validity and reliability scores would strengthen the assessment package and 
balance any weak measures that were to become part of the evaluation package. 

In addition to improving family functioning, short term program staff frequently help 
families mobilize community and professional resources that will adequately meet their needs; 
therefore, the selection of an appropriate social support measure was necessary. While studies 
show a positive correlation between social support and physical and mental health, child 
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development (Cleary, 1988), and parenting adjustment (Turner & Avison, 1985), the practical 
relevance of these findings is as limited as the availability of adequate social support measures. 
Cleary (1988) attributes poor theoretical-practical linkages to the failure of researchers to 
adequately define and measure support concepts and to understand how certain aspects of social 
support contribute to the achievement of treatment objectives. In his report on seven 
instruments. Tardy ( 1985) encourages his readers to adequately define various aspects of 
support. Both authors recommend that instrument questions be carefully examined and selected 
according to their program usefulness. 

Several strong measures were disregarded during the student's search because of their poor 
fit with program objectives. For example, Procidano and Heller's (1983) Perceived Social 
Support Scale and Turner, Frankel, and Levin's (Corcoran & Fischer, 1987) Provision of Social 
Support Scale are valid and reliable measures of familial and peer support; however, they do not 
measure other kinds of support that program families might receive or find helpful. Other 
measures, such as Kaplan's vignettes (Turner & Avison, 1985) do not adequately distinguish 
among support aspects. Based on the program's need for an instrument that would succinctly 
measure the source, frequency, and adequacy of support received by families, Dunst, Trivette, 
and Deal's Personal Network Matrix (PNM) ( 1988) was selected. This instrument (see 
Appendix D) can be regularly used by practitioners to assess and record family needs, current 
and available supports, the frequency of contact with these sources, and how often these sources 
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can be depended upon for support. Blank spaces on one scale allow practitioners and families to 
list family needs and sources of support, while two self-report scales measure the frequency and 
dependability of familial, community, and formal supports. Designed as an assessment device, 
this instrument promotes an understanding of the complexity of social networks and allows 
clients and workers to examine means of activating supports. Unfortunately, the authors do not 
include validity and reliability values. Further, this type of assessment requires considerable 
time to complete. Notwithstanding these drawbacks, worker-client discussions on family needs 
and resources are an integral part of the assessment and treatment process in the short term 
program, so that the PNM could be smoothly incorporated into this process. 

The third aim was to identify measures that could assist program specialists with treatment 
planning and implementation, and that fit the program's behavior-centered Teaching Family 
Model of service delivery. The Goal Attainment Scaling and Family Goal Recording 
instruments appeared to fit with program needs. First, the Goal Attainment Scaling was 
originally developed by Kiresuk and Sherman for use with mental health clients, and is widely 
used in a number of treatment settings, including the Homebuilder's project (Kinney, Haapala, & 
Booth, 1991 ). The measure is designed to provide feedback on client progress toward 
individualized goals. During the initial assessment phase, family members work with specialists 
to establish goals. For each goal area, families then describe current presenting behaviors and 
their rate of occurrence (in percentage form), followed by a list of 2-5 desirable and undesirable 
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outcomes and their rates of occurrence. Each set of projected outcomes corresponds to a 
5-point scale ranging from "Best anticipated success" to "Most unfavorable outcome likely". 

The goal areas can be reviewed as often as needed to guide treatment and determine progress. 
This measure has several advantages: It is flexible, contextual, and focused; provides families 
with realistic expectations of their progress; and enables practitioners to track client change, 
remain accountable, and gather consistent, regular information. It is also important to note that 
the usefulness of this measure is largely determined by the way in which it is used. For example, 
the established goals may be unrealistic, there may be disagreement over which goals and 
problems are considered important, and the wording used may be subject to different 
interpretations. To minimize these problems, clear procedures and guidelines should be 
established prior to instrument use. 

The second promising measure of family progress toward goals was the Family Goal 
Recording (FGR) (see Appendix E for reproduction) proposed by Fleuridas, Rosenthal, Leigh 
and Leigh (1990), which is a modification of the Goal Attainment Scaling. The most significant 
change is that each goal area is assigned a percentage value by family members to signify its 
level of importance. The original measure was revised to allow its users to make regular 
quantitative comparisons within subsystems and to accommodate a family systems approach. 
The authors found the new measure to be reliable and valid when used as part of an outcome 
evaluation package. The FGR can be used by practitioners to track clinical and outcome data. 





and to determine the interplay of variables. While the validity and reliability of the instrument 
cannot be firmly established, it was felt that the education of staff in the use of the measure and 
the use of other standardized measures would balance any weaknesses. The use of this measure 
as part of the treatment planning and intervention process was intended to increase specialists' 
and families' investment in the evaluation process and increase treatment effectiveness. 

Each of these outcome evaluation measures and their corresponding focus area and 
implementation schedule is summarized in table form below. 

Table 1 

Description of Program Outcome Measures 



Instrument 



Focus Schedule 



1. McMaster Family Assessment Device (FAD) Family functioning Case opening/closure 

4-month follow up 

2. Personal Network Matrix (PNM) Social support Case opening/closure/ 

as needed/4-month 
follow up (2 scales) 

3. Family Goal Recording (FGR) Goal attainment Case opening/closure/ 



biweekly/as needed 
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Considerations for the Implementation of Evaluation Approaches 

Having identified the evaluation package, it was important to examine appropriate ways to 
employ these instruments - so they would prove useful. Several methodological issues are 
discussed here. 

First, program staff must be adequately trained in the use of outcome measures (Corcoran & 
Fischer, 1987). Specifically, practitioners need to be knowledgeable in the way instruments are 
presented, administered, scored, and interpreted. This includes an understanding of the 
rationales that can increase staff and family investment, such as enhanced clinical 
decision-making, quality assurance, staff accountability, and ethical conduct. 

Second, it is important to determine who will collect the data (Corcoran & Fischer, 1987). 
Self-report measures can be completed by program staff, family members, or significant others. 
Some authors (Wasik, Bryant, & Lyons, 1990) even suggest that data gatherers and respondents 
share the same culture and racial backgrounds. In the short term program, it was important that 
specialists record the information, since families vary in their ability to understand written 
material without assistance and to follow through with responsibilities; they may respond 
differently to less familiar people; and specialists would want to work collaboratively with 
families and significant others to establish goals or modify family information to accommodate 
professional opinions and concerns. 

The third issue concerns the frequency and consistency with which the measures are 
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administered. Authors concur that regular assessments enable one to plan intervention 
strategies, focus on different family members, engage families in the treatment process, track 
client change, and determine the effectiveness of intervention techniques (Wasik, Bryant, & 
Lyons, 1990). Specifically, Corcoran & Fischer (1987) recommend that measures be issued no 
more than twice weekly to avoid client burnout or over reliance on instruments, and that Rapid 
Assessment Instruments be issued on a weekly basis. In the short term program, specialists were 
to adhere to the following schedule (see Table 1 ): To gather necessary pre- and post- outcome 
data on client progress in the short term program, the FAD and PNM would be issued at case 
opening and closing; to provide specialists and families with clinical information during the 
intervention, the PNM will be issued as needed; to provide information on long-term outcomes 
and to do so in a way that will maintain staff and client investment and cooperation, the easily 
and quickly-administered FAD and 2 of the 3 PNM scales would be issued at 4-month follow up; 
the FGR could provide specialists and families with regular feedback to assist them with meeting 
treatment goals, and would therefore be used at case opening, closing, and biweekly or as needed 
during each client intervention; and to provide target groups with a comprehensive picture of 
family outcomes in this family preservation program, staff would continue to collect child 
placement data at bimonthly intervals over a 1-year follow-up period. 

Fourth, the setting has been identified as an important consideration (Corcoran & Fischer, 
1987). Problems that are setting-specific should be assessed in the corresponding environment. 
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while more general problems can be measured where convenient, providing the locale remains 
consistent. For home-based specialists and families, the home environment may be well suited 
because it is a convenient and frequented location, important family transactions and problems 
are likely to occur in this context, family members generally feel at ease on their own "turf, and 
sensitive issues can be openly discussed and without the risk of being observed or overheard by 
outsiders. 

Finally, different outcome measures should be combined to maximize effective 
decision-making. Corcoran and Fischer (1987) recommend the combined use of self reports, 
observations of client behaviors, and client-kept records to provide workers with different 
perspectives and increase the accuracy of data collection and interpretation. In the short term 
program, current measures include self reports, informal observations by program staff and other 
professionals, and specialist contact notes. It was anticipated that the consistent, regular use of 
the new outcome measures would strengthen existing practices. 

Proposed Solution Strategy 

It was decided that the above three measures (the FAD, PNM, and FGR) would be introduced 
for use in the short term program, based on the aforementioned program needs. Several 
specialists were approached, and stated their preference for the new measures over the current 
ones. To ensure the appropriate utilization of these measures, training was to be provided to 
staff in the administration and scoring of the instruments, in their significance to practitioners 
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and families, and in the importance of their regular, consistent use across program families. 
Policies, client consent forms, and termination format letters would be modified to 
accommodate the use of these measures and the documentation of results, and I would 
collaborate with senior management staff to devise a means for recording and disseminating 
outcome data to program staff and other target groups. In order to measure changes to staff 
evaluation practices, information would be collected from client files and the program's 
statistical records at the beginning and end of the implementation period, and specialists would 
be asked to complete the Specialist Questionnaire at the end of the implementation period. 

It was predicted that client sample sizes and the number of pre- and post- client assessments 
performed would be significantly restricted by the timelines of the practicum project. To 
compensate for this, the specialists would be instructed to conduct assessments appropriate to 
their stage of involvement with clients. For example, specialists who were midway through 
treatment or were closing with families would be required to issue these measures. It was felt 
that these measures would provide valuable information at any stage of the intervention. Finally, 
specialists would still be required to request consent from families to participate in research and 
complete the Parent Child Behavior Questionnaire currently in place. When specialists and 
families began to feel more invested in the evaluation process, they would likely be more willing 
to contribute to the program's development. 



O 

ERLC 



43 



43 



Tea Week Plan 

The proposed 10-week plan was as follows: 

Weeks 1 and 2: First, a clause is added to the "Consent to Participate in Counselling" form 
to allow staff to collect family information using the new outcome measures; and termination 
format letters are modified so specialists can include statistical and clinical outcome data in their 
Child Welfare reports. Next, the new measures are copied, packaged, and stored in the program 
forms file for easy access. Finally, the staff group receives training in the use of the new 
measures, including their implementation, scoring, and interpretation. Time permitting, the 
trained specialists begin to implement the new outcome measures. 

Week 3: Consultation with the program statistician takes place to determine the importance 
of program research endeavors. This information is shared with the specialists, who are then 
encouraged to issue "Consent to Participate in Research" forms to case families at case opening, 
and to issue the PCBQ to consenting families at case opening and closure. Specialists continue 
to implement the new measures. 

Week 4: In order to establish a baseline for evaluation practices, family files and program 
records are examined to obtain information on the number of families that have agreed to 
participate in program research, the number of specialists who have utilized the CBCL and 
PCBQ, and the frequency and consistency of data collection by specialists, over the last 6 
months Specialists continue to implement the new measures. 







‘i 



44 



Week 5: In order to establish a baseline, records of staff contacts with follow-up families 
are examined to determine the consistency with which child placement data has been collected 
over the past 6 months, in addition to the content of the data collected. Specialists continue to 
implement the new measures. 

Week 6: A meeting with senior staff takes place to discuss improvements to data 
collection procedures involving follow-up child placement data. Also, the development of an 
evaluation report book that collectively presents the data compiled on child placement, family 
functioning, social support, and family skills is proposed; and the important targets of 
dissemination are ascertained. Specialists continue to implement the new measures. 

Week 7: Efforts focus on the development and refinement of an evaluation report book, and 
the expansion of program assistant roles to include the regular collection of child placement data 
at follow up. Specialists continue to implement the new measures. 

Week 8: Early data obtained from the Specialist Questionnaire and program files are 
organized into table form. The compilation of information related to the use of the three new 
outcome measures begins. Specialists continue to implement the new measures. 

Week 9: The Specialist Questionnaire is issued to specialists on Monday, and the 
completed forms are collected on Friday. Specialists continue to implement the new measures. 

Week 10: Pre and Post results taken from the Specialist Questionnaire, program records, 
and client files are compiled, contrasted, and presented in table form; and statistics on the past 





and current implementation of the research-based PCBQ are compared. The results are shared 
with the staff group. 

It was anticipated that several factors could interfere with the above activities. First, staff 
might have decided to book holiday time, which would limit the number of staff available for 
scheduled meetings and the number of specialists present to regularly implement outcome 
measures and complete the Specialist Questionnaire. To compensate for this, 2 weeks would be 
reserved in addition to the 10 weeks allotted for the implementation phase. Also, specialists 
covering for absent case specialists would be required to perform family assessments using the 
new measures, so that these results could be shared with each case specialist and included on the 
Specialist Questionnaire. If one Program Director was unavailable for staff meetings, the second 
Program Director or another authorized person had agreed to be present in his place. Finally, 
vacation schedules would be consulted and the Specialist Questionnaire issued to on-leave 
specialists at an earlier date, should their absence coincide with the pre-scheduled administration 
of the questionnaire. 

The activities were to be tracked using a weekly record sheet with corresponding check boxes 



and spaces in which to comment on the process. 
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CHAPTER 5 
RESULTS 

The aim of this practicum was to develop and implement appropriate evaluative outcome 
measures in the short term program. The literature findings and practitioner information 
presented earlier substantiate the need for practical, comprehensive measures; the consistent, 
timely administration of these measures by trained staff; and an outcome data package that 
provides useful information on client change to a wide range of staff and consumer groups. 

To attain the project goal, this information was used to formulate objectives and guide the 
implementation of specific solution strategies over a 12-week period. This chapter includes a 
summary of variations to the proposed strategies, a presentation of project results and 
explanations for their presence, and a discussion of key problems as they relate to the present 
and future attainment of project objectives. 

Implementation 

While the implementation plan outlined in Chapter 4 was essentially followed, several 
anticipated and unanticipated variations in the timing and outcome of the proposed strategies 
transpired. First, four absent specialists received training subsequent to the staff evaluation 
workshop, several specialist and management meetings were rescheduled when unanticipated 
work obligations took precedence, and three specialists who had scheduled work leaves and 
vacation time completed the post questionnaire 1 month prematurely. To offset the potential 
consequences of these changes, the 10-week implementation period was extended to 12 weeks to 
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provide specialists with every opportunity to utilize the program measures and receive feedback 
prior to the collection of post-test data in Week 12. Second, substantial program funding cuts 
by Social Services forced the elimination of the Program Assistant role, which meant that the 
assistants would not be accountable for the collection of child placement data, as was originally 
proposed. To compensate for this, specialists were deemed responsible for collecting this data 
from their former clients, which 1 volunteered to collate and disseminate every 3 months along 
with 3-month follow-up data on family functioning. While budget cuts served to highlight the 
need for improved evaluation practices, the loss of program staff (Program Assistants, the 
Trainer/Evaluator, clerical staff, and the statistician), an upcoming increase in specialist 
caseloads, and an increase in specialist evaluation responsibilities has placed added demands on 
staff resources. Third, to increase the sample size, it was decided that the Native Program 
(consisting of two staff members) would also participate in the project, and that the data would 
be included in pre- and post-test results. Finally, meetings with the program statistician, agency 
staff, and outside professionals failed to yield any strong rationales for the continued use of the 
Child Behavior Checklist (CBCL) and the Parent Child Behavior Questionnaire (PCBQ). 
Nevertheless, staff were encouraged to collect PCBQ data, so it could be used together with 
program data to identify family functioning patterns. 

In terms of the post-test results, these factors could have had an impact on staff investment 
and follow through Other possible explanations are presented in conjunction with project 
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results. 

Findings 

The first practicum objective was to select outcome measures that would provide specialists 
with clinical feedback on family changes and progress toward treatment goals. During the 
evaluation training workshop, criteria and rationales for the selection the Family Assessment 
Device (FAD), Personal Network Matrix (PNM), and Family Goal Recording (FGR) as 
replacement measures were provided to staff, who were then directed to implement the measures 
with case families. The Specialist Questionnaire (see Appendices B and F) was used as a pre- 
and post measure of the appropriateness of program instruments. 

The results revealed that the new program instruments are of greater clinical value than the 
former instruments. Specifically, during the pretest phase 22% of the practitioners reported that 
program instruments were useful for measuring changes that take place within program families, 
while post-test results revealed that 44% found the newly-implemented measures to be sensitive. 
Further, during the pretest phase 33% of staff reported that the outcome data were consistent 
with their own observations regarding family situations and progress, compared to 67% at 
post-test. These data are highlighted in Table 1, along with percentages constituting unfavorable 
staff responses and the absence of responses (symbolized by "NA"). (Note: Table categories 
were created by collapsing data that did or did not appear in the first and second, or third and 
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Table 1 

Practitioner Responses to a Pre- and Post Questionnaire 



Items 


n 


% Responses 

Favorable Unfavorable 


NA 




Appropriateness of Measures 






1 . Sensitivity of instrument 


Pre 


9 


22 


67 


11 


Post 


9 


44 


- 


56 


2. Data consistent with 
workers' observations 


Pre 


9 


33 


22 


44 


Post 


9 

Adequacy 


67 

of Staff Skills and Training 




33 


10. Statistical analysis of data 


Pre 


9 


33 


67 


- 


Post 


9 


78 


22 


- 


11. Clinical analysis of data 


Pre 


9 


56 


44 


- 


Post 


9 


78 


11 


11 


12. Adequacy of training 


Pre 


9 


11 


89 


- 


Post 


9 


78 


11 


11 
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At post-test, several of the specialists who answered NA commented that they have not had 
or taken advantage of opportunities to use the measures with case families. In part, this could be 
due to the fact that actual case openings outnumbered the closings and follow ups, which 
restricted the pre and post use of the measures. Also, the benefits of data collection were limited 
in cases where specialists were late into their interventions. In these instances, the majority of 
clinical decisions had already been made, so that additional information provided by the 
measures would not have been useful. Of the specialists who did use the measures, the FAD and 
PNM were cited as those used most frequently (only one formally reported using the FGR). This 
is probably because of the ease with which the scales are administered and interpreted. The 
time-consuming nature and complex presentation of the FGR (see Appendix ) could in part 
account for its limited use. Had the FGR been used by more specialists, they would have been 
able to track client progress more frequently during the implementation phase. Indeed, one 
specialist informally commented that the FGR was a valuable and sensitive measure that 
facilitated progress in a "difficult" family. Notwithstanding this feedback, the staff polled at 
post-test either felt positive or uncertain about the new measures, while pretest results revealed a 
predominantly negative view of program measures. 

In summary, although the results are less significant than was anticipated, one can conclude 
that the first objective has been partially met. 

The second objective was to have the practitioners become more familiar with and skilled in 
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the implementation of outcome measures and the interpretation of outcome data. This entailed 
training staff in administration and scoring procedures, and initiating contact with staff to answer 
questions or update them on recent modifications to program evaluation practices. Again, the 
Specialist Questionnaire was used as a pre- and post measure of improvements to staff . 
familiarity and competence (see Table 1). The data revealed an increase in specialists' 
familiarity with the use of evaluation measures. Respectively, 33% and 56% of specialists 
initially reported an ability to statistically and clinically interpret test data, compared to 78% 
during the post-test period. Further, evaluation training was deemed adequate by 1 1% of pretest 
staff, compared to 78% at Week 1 2. 

Staff commented that while training was instrumental in developing competence, the 
practical application of the measures constituted a significant part of the learning process. This 
would explain why most of the specialists who rated their skills and training positively had 
utilized program measures at some point during the implementation period. The fact that 
specialists had no previous experience in the statistical and clinical interpretation of program 
outcome data would also explain why training by itself has failed to significantly increase the 
reported skill level of some specialists. Of those who rated their skills and training unfavorably, 
most commented that they felt confident their skill level would increase with instrument use. 
Again, of the three current measures, the FGR was the least familiar to staff, even though it 
requires the evaluator to perform simple calculations and design goals using basic interview 
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skills. It is possible that since all practitioners are required to formulate comprehensive 
treatment plans with their clients, the FGR was viewed as a redundant exercise. Further, one 
specialist attempted to use the measure commented that it did not "fit" well for a particular 
family. Again, poor investment in the use of the FGR could account for its limited use, which in 
turn could affect specialists' perceived level of competence and degree of familiarity. 

Taken together, these outcomes reveal that the second objective has been partially met. The 
data on the adequacy of training are particularly significant and promising. 

The final objective was to increase specialists' use of program measures. This translates to an 
increase in the number of specialists using the measures, an increase in the consistency with 
which interval testing is conducted with case families, and an increase in the frequency of 
testing. Progress toward this three-part objective was measured using the Specialist 
Questionnaire and client file data. (Note that only cases that had opened, closed, or been closed 
for 3 months during the implementation period were included as client file data.) As part of the 
solution strategy, in-service training was offered to supervisors and front-line workers in order to 
maximize staff accountability and participation at all levels. These participants received verbal 
instruction, written information, examples of instrument use and scoring, and timetables 
depicting schedules for the use of program measures. Written and verbal reminders regarding 
instrument use were also issued to front-line staff, and supervisors were asked to ensure 
practitioner follow-through during their supervision consultations. To facilitate data collection 
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by staff and to enhance the acquisition of client file data for this project, each practitioner was 
provided with a booklet in which to record and store outcome data and program documents. In 
addition, a clause was added to the "Consent to Participate in Counselling" form (see Appendix 
G) to allow all practitioners to collect outcome data as part of the treatment process. Staff were 
also encouraged to obtain client consent to pa r xipate in research. 

In response to the first part of this objective, client record data revealed that 82% of 
specialists sampled had obtained at least one written client consent during the 6-month pretest 
period to use outcome data in program research. Seventy-one percent of this group took 
advantage of the opportunity to implement program instruments. This constituted 59% (« = 17) 
of the entire specialist group. Conversely, by the end of the 12-week project, the new 
counselling consent forms were obtained by only 46% of all specialists; 54% («=13) of the entire 
group had used one or more of the new measures on at least one occasion; and signed research 
forms were obtained by 77% of specialists. This would suggest that the number of specialists 
using program instruments has neither increased or decreased, that the number who obtained 
authorized counselling forms was insignificant, and that the number who received authorized 
research forms has not significantly changed. 

There are several possible reasons for this. First, one specialist reported that she carried no 
case families during the study period, while others explained that they were midway through 
client interventions, and therefore did not feel these measures would benefit treatment. Since 
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these measures were promoted primarily as clinical instruments and secondly as research 
instruments, it is likely that the costs to specialists (for example, the poor timing of the clinical 
instruments and the effort and time commitment involved in their use) outweighed the perceived 
(research) benefits. The brief duration of the implementation phase would further support this 
type of reasoning. Another explanation is the absence of consistent follow-through at the 
supervisory level. One supervisor and one specialist commented that the majority of program 
coordinators (with the exception of my supervisor) did not strongly encourage or monitor the use 
of the measures among their supervisees. Finally, some copies of the outdated consent forms 
remained in circulation, and were being used in place of the revised ones. The fact that 
specialists collected outcome data without securing the proper consent, and that others received 
client consents but did not implement the new measures with these families indicates that the 
use of counselling consent forms neither created or detracted from specialists' opportunities to 
conduct client evaluations. Taken together, the results do not imply that staff lack good 
intentions or insight into the importance of program evaluation: On the contrary, several 
workers commented that they planned to use the measures once they have opened with new 
families, and 67% of specialists acknowledged that the new measures were beneficial to the 
program, compared to 1 1% at pretest. 

In summary, the number of specialists using program measures has failed to increase. 
Therefore, this portion of the objective has not been satisfactorily met. 
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Regarding an increase in the consistency and frequency of outcome testing, staff adherence 
to the evaluation schedule is presented in Table 2, in the form of staff responses to the Specialist 
Questionnaire, and the actual number of consent forms and instruments completed by 
practitioners with case families. To reiterate, counselling forms were modified to facilitate the 
clinical use of measures and increase the likelihood that program participants would agree to 
participate in program research endeavors if direct treatment benefits were available. In 
addition, staff were told that the PCBQ might provide useful information when used in 
conjunction with the FAD, even if professionals could not justify its use. Finally, the follow-up 
interval was shortened from 4 months to 3 months to coincide with the collection of child 
placement data, and to increase the likelihood that former clients could and would be contacted. 
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Table 2 

Specialists' Actual and Reported Adherence to Evaluation Pre and Post Schedules 



Items 


% Adherence 
Reported n 


Actual 


n 


3. Request written consent 


Pre 


75 9 


64 


61 


Post 


72 9 


22 


32 


4 Issue measures at case opening 


Pre 


33 9 


20 


61 


Post 


56 9 


25 


28 


Issue measures at case closure 


Pre 


11 9 


1 


36 


Post 


22 9 


21 


19 


6. Issue measures at follow up 


Pre 


0 9 


0 


14 


Post 


0 9 


0 


7 
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The consistency of measurement use is addressed on two levels: The overall consistency of 
measurement use (reported and actual) among specialists within each test interval , and the 
consistency of measurement use within family interventions. Pretest data depict a lack of 
consistency regarding overall staff adherence to evaluation schedules (see Table 2). 

Specifically, staff obtained written research consent from 64% of their clients, and the measures 
were implemented only 20% of the time at case opening, 1% at closure, and 0% at follow up. 
Although the figures are modest, the post-test staff group was generally more consistent in their 
follow through. Specifically, 22% of client counselling authorizations were obtained, and 
program measures were issued 25% of the time at case opening and 2 1% at closure. As with the 
pretest group, the specialists collected no follow-up data during this period. Note also that the 
PCBQ was utilized by only one specialist from the post-test group. While specialists rated their 
follow-through much higher at pre and post than was actually the case, they were aware that 
their adherence to the evaluation schedule tapered off from case opening to case follow-up, 
which was the actual trend described above. This discrepancy between self reports and client 
file data appears greatest in the first half of the schedule. 

In terms of specialist consistency within client cases, specialists issued at least one of the 
measures according to the schedule 9% of the time at pretest, compared to 28% at post-test. The 
FAD and PNM were used as required by most of these post-test workers. The measures were 
most commonly used at case opening and closing, less commonly used midway through 
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interventions (this is tabled under case opening), and not at all at case follow-up. 

Several explanations for these trends are possible. First, many staff explained that they 
received verbal consent from clients to use the new measures, which could account for the 
discrepancy between the number of reported requests for consent and the actual number of 
consents obtained. Also, the specialists who did not complete and return the Specialist 
Questionnaire might have been among those who did not use program measures. Had their 
responses been included, it is possible the reports would more closely match the actual results. 
Second, perhaps staff based their self-reports on the number of perceived opportunities versus 
actual opportunities to use the measures (recall the cost benefits described earlier). For example, 
if it was felt that the measures would not benefit the treatment of a particular family, this may 
not be viewed by specialists as an opportunity. 

Third, the lack of consistency within case families at post-test could stem from the fact that 
specialist were nearing the end of an intervention, and decided against testing until case closure. 
These late interventions were classed as opportunities («) under the case opening results, which 
may have had a negative effect on the project results. Fourth, while staff were provided with 
rationales for the consistent use of the measures, they expressed their reluctance to contact 
former families. Their lack of motivation and not the decline of families could therefore account 
for the continued absence of data collection at follow up. Indeed, 56% and 44% fewer staff 
attributed families' reluctance to participate in research to uncertainty around their commitment 
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and the absence of treatment benefits, although program confidentiality and ethical assurances 
remained unchanged at 56%. Fifth, given the absence of rationales for and lack of training 
provided in the use of the PCBQ, and the priority given to the piloting of new measures, it is not 
surprising that only one specialist completed the PCBQ. 

In summation, there was a small yet promising increase in the consistency of measurement 
use, especially within case interventions. Therefore, the second part of the objective has not 
been adequately met. 

These same results look different when the frequency of instrument use among specialists is 
examined (see Table 2). It has already been surmised that requests for written client consent and 
family cooperation are not strongly related to the consistency and frequency of instrument use by 
specialists. Notwithstanding the decrease in requests for consent during the post-test phase, 
there was no substantial change in the frequency of testing at case opening (20% versus 25%), 
closure (1% versus 21%), and follow up(0%) during the implementation phase. (Note that prior 
to calculating percentages, the completion of a single pretest instrument per child and a post-test 
instrument per family was valued at 0.5, and the completion of two or more measures was 
valued at 1 .0.) The most impressive increase occurred at post-test, when instruments were 
administered to 20% more families at case closure than during the pretest interval. Again, 
together staff rated the frequency of instrument use higher than was actually the case. Not 
surprisingly, the majority of staff (78% at pretest and 67% at post-test) identified work demands 
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as the primary obstacle to instrument use. This is in spite of the fact they feel the new 
instruments contribute valuable information to the program and to program research. Again, it 
would appear that the perceived costs (time, energy, and the timing of the interventions) 
outweigh the benefits (enhanced clinical decision making and program development) of frequent 
instrument use by specialists. 

The results show no significant increase in the frequency of instrument use, meaning that this 
portion of the objective has not been attained. This in conjunction with the aforementioned 
results suggest that although increased measurement use by specialists was anticipated, it did not 
occur. 

Finally, although it was not formally included as a project objective, attempts were made to 
improve the use of outcome data by program staff. It was felt that this would contribute to staff 
satisfaction with evaluation changes. First, staff were encouraged to share outcome data with 
other case professionals. In addition, a final sentence was added to the Termination Format 
Letter (see Appendix H), prompting staff to use statistical and clinical data to substantiate the 
assessments, treatment approaches, and recommendations described in their reports to Child 
Welfare. 

According to data gathered from the Specialist Questionnaire, there was a small increase in 
specialists' use of program outcome data. Specifically, at pretest 0% of workers stated that they 
used the data to enhance their clinical decision-making, compared to 22% at post-test. Further, 
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0% indicated that they share the data with other professionals, while 33% did so at post-test. Of 
those who did not make use of the data or who have not yet collected data, most explained that 
they intend to do so in future. Only one worker reported that the FAD (alone) produced 
insufficient data. Again, it is likely that specialists will develop confidence in using and sharing 
data once their familiarity with the measures increases. Earlier findings suggest that specialists 
have not yet reached this level of confidence. 

Next, the development of a comprehensive data package including child placement and 
family functioning data was introduced for discussion at management meetings and was later 
shared with the staff group. Because the collection of child placement data was inconsistently 
collected (it was more than 6 months behind), it was suggested that Program Assistants 
permanently assume this responsibility. The Program director instead elected specialists 
responsible for collecting this tn-monthly information from their former case families. (It was 
later discovered that the Program Assistant positions were being eliminated.) To address the 
problem of narrow child placement data, Client Discharge forms were modified (see Appendix 1) 
to include information on the projected future placement of children living in restrictive settings, 
and to help professionals ascertain the favorability of out-of-home placement. Next, two forms 
(see Appendix J ) were created so that child placement and family functioning data could be 
presented collectively to program staff, contract managers, and other professionals on a regular 
(tri-monthly) basis The first sheet includes child placement, reporting period, sample size, and 
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long-range planning information, while the latter sheet displays the reporting period, sample 
size, and client pre- and post-test FAD scores related to seven aspects of functioning. 

Lastly, specialists were asked to gather 12 months of child placement data, and 1 compiled 
and displayed 3 months of recent data on the new form for staff to view. Due to the lag in data 
collection, this sample size was limited to 66 of 143 possible cases, with 6 of them incomplete. 
It is hoped that prompting specialist at more regular intervals will increase future sample sizes. 
Family functioning data will be made available to staff during the next reporting period. It is 
likely this will also increase staff satisfaction with the outcome evaluation package and the 
consistency of feedback. 

As a result of these recent changes, the number of positive staff ratings regarding the 
consistency and frequency of data collection increased from 1 1% to 56%; the appropriateness ol 
child placement data and the outcome data package increased from 22% and 0% to 44% and 
67%; and staff ratings of their participation in data collection was 1 1% higher. Further, staff 
reported feeling slightly less pressured to ensure in-home child placement. Together, these 
results suggest that staff feel more positive about the collection and use of child placement data. 

It is important to note that while the size of the pretest respondent group was modest, the 
post-test group matched it in size and included many of the original specialists (even though 
fewer specialists returned the questionnaire at post-test). It is hoped that this will increase the 
validity of project results. In addition, a total of three client files (two at pretest and one at 
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post-test) were missing from the program storage units, and were therefore not included in the 
results. The large client data sample would likely compensate for these omissions. Finally, 
pretest client file data was gathered over a 6-month period, while the implementation phase 
covered only 3 months. Had the implementation period been 6 months in duration, it is felt that 
staff ratings and client file data would reveal an increase in measurement use. 

Discussion 

Despite a modest increase in specialists' familiarity and satisfaction with program measures, 
the overall results of this study, particularly as they relate to instrument use, were unimpressive. 
While explanations for the negative and positive results have been offered in the context of 
project objectives, this discussion will focus on those having the greatest impact on evaluation 
practices in the short term program. 

First, it is believed that the limited duration of the implementation phase significantly 
impacted the post-test results. While a concerted attempt was made to alter staff practices and 
guard against anticipated problems, changes did not occur quickly or in the absence of 
difficulties. Had the implementation period been at least equal to the 6-month pretest period, 
staff ratings and client file data might reveal an increase in measurement use, as specialists' 
familiarity with the measures, opportunities to use the measures, and chances to work through 
the problems discovered during this study increased. This proposed time frame is reasonable, 
since the evaluation patterns measured in the pretest phase had 2 years in which to evolve. 
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Two other problems identified in this study should be addressed before the upgrading of 
program evaluation practices continues. The first relates to the attitudes of program staff 
surrounding program evaluation in general and this pilot project in particular. It was noted 
earlier that many specialists did not take advantage of opportunities to use the new measures, 
and that their regular use was not enforced,^ the management level. Workers and supervisors V 

do not appear to view evaluation as a priority undertaking, despite their enthusiasm and good 
intentions. This needs to change, or future improvements to program evaluation will not have 
the desired impact. 

A second and related problem involves increasing staff workloads. Durino this study, the 
program budget was substantially reduced, administrative and support staff positions were 
eliminated, and specialist case loads and paper workloads increased. During the pretest phase, 
the majority of staff identified work demands as a primary obstacle to instrument use. It stands 
to reason that this problem also existed during the post-test interval. Therefore, there is a need to 
locate and devote more resources to evaluation duties. 

Although the introduction of appropriate measures, in-service training, and the use of a 
comprehensive data package yielded positive ratings from staff group, the pervasive nature of 
the above problems calls for creative solutions that can shift the cost-benefit ratio of program 
evaluation in the desired direction. In Chapter 6, the implications of these results and the future 
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CHAPTER 6 
CONCLUSIONS 

In this chapter, evaluation problems of relevance to family support programs are addressed, 
and the importance of evaluation in future is discussed. 

During the course of this project, attempts were made to upgrade evaluation procedures in 
the short term program. Through careful research and planning, it was hoped that most of the 
barriers to outcome evaluation inherent in the field, family support programs, and the short term 
program would be alleviated. The modest project results described in Chapter 5 highlight both 
the promise and problems associated with program evaluation. It was learned that while staff 
familiarity with program measures, their satisfaction with the outcome data package, and the 
consistency of assessments within client cases modestly increased, the number of specialists 
using program measures and the overall consistency and frequency of client assessments 
remained unchanged, except for a small increase in the number of assessments performed at case 
closure. While staff identified competing work demands, limited opportunities for the clinical 
use of program instruments, and lack of support and monitoring at the supervisory level as the 
primary obstacles to instrument use, they acknowledged the importance of program evaluation 
and research and concurred that the new measures promoted competency and yielded valuable 
feedback. 

The positive results would suggest that the selection of comprehensive and relevant measures 
and training in their use have diminished some of the problems attached to program evaluation. 
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However, the failure of this pilot to increase the frequency of instrument use among all 
specialists and across all stages of client intervention suggests that for specialists, the costs have 
outweighed the benefits of program evaluation. 

Recommendations 

In Chapter 5, the main obstacles to instrument use were identified as the restrictive timelines 
of the project, the failure of staff to treat program evaluation as a priority undertaking, and 
limited program and staff resources. It is hoped that the following recommendations will yield 
positive results in future for the short term program, as well as for family support programs faced 
with similar challenges. 

First, it is likely that the results of this project were not captured in their entirety because new 
evaluation patterns had not yet been fully formed at the time of the post-test. Nevertheless, the 
project results accurately depict the primary obstacles to meeting project objectives, indicate 
some desired change in the general direction of positive practices, and shed some light on the 
relationship between study outcomes and a number of variables. Indeed, other programs would 
be wise to follow suit and assess the effectiveness of particular strategies early in the pilot phase, 
so they can make necessary changes before a decision regarding the effectiveness of new 
approaches is reached. Further, once evaluation patterns have had time to stabilize, a final 
assessment would provide a more accurate picture of project outcomes. For the short term 
program, this means that the information learned in this study should be used to further upgrade 
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program evaluation practices, and that another assessment should be performed once these 
strategies have had an opportunity to fully impact evaluation practices. By viewing this study 
period as a first step to improving program evaluation, objectives can continue to be met. 

Second, staff identified their heavy case and paper workload as a primary obstacle to 
instrument use. Wasik, Bryant, and Lyons (1990) cite this as a common problem in the social 
services field, and warn that it frequently leads to worker "burnout". One main culprit, shrinking 
program budgets, can have a negative impact on staff resources, as well as on quality assurance 
and staff morale. For example, the Prenatal/Early Infancy Project employed home visiting 
nurses to provide health and other support services to disadvantaged pregnant mothers. When 
program funds were slashed and case workloads increased, the nurses reported that their "...work 
seemed less valued and [they] felt they couldn't give their clients the kind of support they 
required" (Schorr, 1989, p. 174). Dissatisfied, these nurses eventually left the program. In 
addition, it was no longer possible for the program to finance evaluation studies to determine the 
effectiveness of these step-down services. 

To buffer against these negative effects, programs must find ways to reallocate existing 
resources, so that service quality remains high and evaluation practices can continue to be 
upgraded. Similarly, the short term program needs to examine ways in which its resources are 
currently being used. For example, specialists claim to spend a significant portion of their time 
documenting client interventions. Wasik, Bryant, and Lyons ( 1990) warn against too little or too 
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much documentation, and explain that the latter can result in poor quality records. Cole and 
Duva stress that documentation should be "...kept to a minimum.. .and not [be] too duplicative" 
(Cole & Duva, 1990, p. 52). Several programs have already attempted to solve the problem of 
limited resources: Some Calgary-based social workers audiotape their meetings and submit the 
narratives to clerical staff for typing, while the Infant Health and Development Program (Wasik, 
Bryant, and Lyons, 1990) uses standard check list summary forms so workers can briefly record 
pertinent information on their professional and client contacts. This information is then stored in 
a computer and used to track individual client progress and the effectiveness of various sites. 

For family specialists in the short term program and other programs, further examination and 
streamlining of documentation procedures would be helpful. Indeed, a program technology 
meeting has been scheduled to discuss ways in which staff can deal with the loss of personnel 
and cope effectively with their increasing job demands. This is an appropriate forum in which to 
discuss the need for evaluation resources 

In retrospect, the problem of increasing work demands was probably exacerbated by the 
sudden introduction of three new instruments to the program at the beginning of this study. The 
introduction of one measure at a time for program use may have yielded better results, as staff 
found ways to gradually incorporate these responsibilities into their work routine. 

A third obstacle to evaluation follow-through is the way staff view program evaluation. 

While staff in the short term program recognize its value, they do not treat outcome evaluation as 
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a priority. This was evident at both the supervisory level and front-line level. As was mentioned 
earlier, budget cuts and increased workload demands can leave staff feeling devalued, 
disempowered, and unable to effectively perform their responsibilities. Because evaluation 
responsibilities were carried out in the past whenever resources became available, it is 
understandable that other work responsibilities would continue to take precedence over 
evaluation. 

There are several ways to engage staff in the evaluation process. One is to increase the level 
of support and monitoring at the supervisory level. If coordinators expect staff to take evaluation 
seriously, they must use their leadership skills to encourage and motivate front-line staff. 
Everyone must feel that evaluation is a shared responsibility. A second suggestion is to allow 
staff the freedom to make more decisions involving evaluation. The majority of the post-test 
group used program measures only when they felt them to be of clinical significance. Those 
who did not may have felt that their choices were too limited. While all staff should use at least 
one standardized pre and post outcome measure (in this case, the Family Assessment Device), 
specialists should be allowed to choose from a battery of instruments, based on their clinical 
decisions, program guidelines, and the individualized needs of their clients. For example, the 
use of Hudson's Multi-Problem Screening Inventory (MPSI) Depression subscale with a 
depressed parent, or the assessment of an adolescent sexual perpetrator using the Wilson Sexual 
Fantasy Questionnaire would allow staff to address the specific problems encountered by family 
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members. Further, by selecting their own measures from an available pool which they help to 
form, specialists are more likely to feel empowered and invested in the evaluation process. 

Recall in Chapter 2 how one local professional remarked that involving practitioners in the 
creation of a program measure increased their investment level, despite the time-consuming 
nature of the instrument. The type of measures used and their popularity can also assist with the 
identification of the clinical populations most frequently served by the program. 

A third way to engage staff is to increase their sense of ownership with regards to evaluation. 
By expanding the formal job description of specialists to include that of evaluator, they are more 
likely to feel this responsibility is a valued and recognized part of their work identity. This will 
likely prove most successful with new recruits, whose expectations of the work role have not yet 
been formed. Two new workers have already received pre-service training in the use of program 
outcome measures, which will further contribute to their investment in evaluation practices. 

It is hoped that the above recommendations will yield positive results in the short term 
program, and contribute to the learning of other professionals. The documents and training 
package introduced during this project have been adopted for use by the program. Also, the 
research outcome data that is being collected quarterly for this fiscal year will be shared with 
professionals from other agencies during the Annual Meeting, and will be presented in an annual 
program report titled "Schedule A" to the Department of Social Services. I will continue to 
make improvements to program evaluation, and plan to conduct another assessment of the pilot 
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once staff have had an opportunity to use the measures and once the program has approved and 
accommodated the proposed changes. 

Final Comments 

The resuits of this project have highlighted the persistent challenges associated with program 
evaluation. Information on barriers such as economic hardship, staff resistance, and the 
magnitude of change required to bring evaluation practices to an acceptable level, serve to put 
the results of this study into perspective and balance the expectations of those attempting to 
improve staff practices. While effective evaluation procedures require considerable time and 
energy, programs should maintain a sense of urgency because history has taught us that time 
brings with it new changes. 

One anticipated change is the era of managed care (Monack, 1995). Gaining popularity in 
the public and private arenas of America, its primary goal is to reduce the cost of services such 
as hospitalized or institutionalized care, and more adequately meet the needs of disadvantaged 
populations at the community level. For example, Medicaid-funded programs will allegedly 
spare federal money and reserve state funds for the continuing development of community 
services. New words such as consumer demand, multidimensional service delivery, and 
regulaied services, replace familiar concepts such as fee-for-service care. Although Canada has 
not yet reached a verdict, the local deterioration of the Alberta Health Care system and the 
closure of three Calgary hospitals makes it likely these neighbors will follow suit. 
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The implication of managed care is that the ".. .demand for community-based services... with 
documented positive outcomes is increasing" (Monack, 1995). As states and provinces pay more 
attention to the services they are purchasing and less to credentialling and service availability, 
programs will need to demonstrate their effectiveness using sound evaluation practices. In so 
doing, they may be delegated as key players and decision makers. 

In summary, even though the results of this study are disappointing, the persistent problems 
and small gains associated with upgraded evaluation practices in the short term program provide 
a basis on which to build future efforts. Aware of the challenges and armed with the means and 
motivation to overcome them, it is hoped that evaluation can eventually serve its intended 
purpose within all family support programs. Efforts to strengthen and substantiate quality 
assurance must continue and must include the creation of work settings that are conducive and 



receptive to evaluation development and practices. 
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APPENDIX A 

Parent-Child Behavior Questionnaire (PCBQ) and Child Behavior Checklist (CBCL) 
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1 . 0 . 



FAMILY: 

NAME OF PARENT FILLING 
OUT QUESTIONNAIRE:. 



DATE: 



Pre 



Post 



18 Week 
Followup 



INTERACTION BEHAVIOUR QUESTIONNAIRE (PCBQ) 



Below Is a list of statements that may or may not be true about (name) _• 

r nn ,<^r tho last four weeks only. For those statements that are mostly true, circle 
YES, for those that are mostly false, circle NO. Try not to leave any blanks. 


True 


False 


1. 


Hy child Is easy to get along with. 


True 


False 


2. 


Hy child Is well behaved In our discussions. 


True 


False 


3. 


Hy child Is receptive to criticism. 


True 


Fal se 


4. 


For the most part, my child likes to talk to me. 


True 


False 


5. 


We almost never seem to agree. 


True 


False 


6 . 


Hy child usually listens to what I tell him/her. 


True 


False 


7. 


At least three times a week, we get angry at each other. 


True 


False 


8. 


Hy child says that I have no consideration of his/her 
feelings. 


True 


False 


9. 


Hy child and I compromise during arguments. 


True 


False 


10. 


Hy child often doesn't do what 1 ask. 


True 


False 


11. 


The talks we have are frustrating. 


True 


False 


12. 


Hy child often seems angry at me. 


True 


False 


13. 


Hy child acts Impatient when I talk. 


True 


False 


14. 


In general, 1 don't think we get along very well. 


True 


False 


15. 


Hy child almost never understands my side of an argument. 


True 


False 


16. 


Hy child and I have big arguments about little things. 


True 


False 


17. 


My child Is defensive when I talk to him. 


True 


False 


18. 


Hy child thinks my opinions don't count. 


True 


False 


19. 


We argue a lot about rules. 


True 


False 


20. 


My child tells me he/she thinks I am unfair. 



HC*J**-ADMW/<NTtAAC OUt-OI/1 WHS 
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CHILD BEHAVIOR CHECKLIST FOR AGES 4-18 



Child* 

NAI4( 



tlx 

o 



D c*i 



AOC 


CThniC 




0*0UF 




OR RACE 



tOOAT* OAT{ 



0|I|. 



OKkOt IN 
SCMOOC 




Piossa fill out Ihla form to rsftict youf 
»iaw of thi chltd*a bohsvior tvan if othor 



not AnewoiNo 
SCHOOt □ 



paopla might not i0rti, F««l frit to writ* 
additions! comments bosids ««ch itsm 
•Ad In tha spacas provided on paga 1 



PARENTS* USUAL TYPC OF WORK, in* H *♦( wwtlng **». /P*, 

•• • »•«#*. fttft* m 4 Ch*A<. Arp* ICNNIMCN/. A#* |m 

tiMnr. tstht IM MiMNA, army ••rfttftf.j 

FATHER* 

TYM Of WORK ___ 



MOTHERS 

TYPE Of WORK 

THIS f ORM flit CO OUT $Y; 

O UolM' fMm#} 

□ f*\h4t pumti . 

Q OtMf- Mm# i f#littO*Pwp to (I# 



t Platt a flit th# sports your child most ffkoo 
1# tiU p«n In. For •Kimpli. swimminQ, 
bastbatf, skstlng, skats boarding. plki 
Tiding, fishing, #tc. 

□ Nono 



A. 



b. 



c. 



Compift d to othifi of Ihs itm« 
iQi, About how much tlms dots 
hi/thi ipmd tn ##ch? 



0#*Y 

Ka#w 


lt#s 

Than 

A«#ugs 


Auriff 


Mwt 

Tha# 

Avififi 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 



Compiled to othifi of Ihi ••m« 



ago, how wall doaa ha/tht 
ona? 


do *s< 


Oonl 

Know 


••tow 

Av#r#f# 


A»#fSf# 


Abort 

Af«ri|t 


O 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


O 


□ 


□ 


□ 



H * **•••# W»t youf child*! flYOflti hobblas, 
•ctlvlitii, md gimii, othir thin sporti. 
For iiampit: stamp*, dolls, books, piano, 
craftt, cars, singing. «tc. (Oo not tncludo 
listming to radio or TV4 

O Non# 



Compand to othsrs of ths ssma 
sgs, about how much Urns dots 
hs/shi spond in ischt 



Know 



ls#s tl#r# 

Than Avtrtf# Tha# 
Anfif# Avltift 



CoMparad to othtu at tba i*m# 
'»♦. how won doat ha/aht 4 c etc. 
on#? 



P##1 

Know 



• •nrw « 

‘""I* 



AMw 

A*#rif# 



□ □ □ □ 

□ □ □ □ 

□ □ □ □ 



□ □ □ □ 

□ □ □ □ 

□ □ □ □ 



lit ?>•••• n.t any organttatlona, atuba, 
Utmt, or group* your child belong* to. 

O Nono 



0. 



€« 



Compsrtd to othan of ths ssma 
aga, how actlva la ha/sha In aach? 



Dwt l#sa 
Know Aathi 

□ □ 

o □ 

□ □ 



Arttw 

□ □ 

Q □ 

□ □ 



IV. ft**** n*t any Job* or chora* your chM 
ha a, for oxampto: papor rout*, babysitting, 
flvahlng bod, working in *tor*. ate. (Includo 
both M<< w*b«id job* and chor**4 
Q Non* 



b. 

c. 



Comparad to othora *1 th* atm* 
aga, how woll doat hafah* tarry 
(ham awtt 



Daw* 

Km 

□ 

□ 

□ 



Ararag* Arwtgt 

□ □ □ 

□ □ o 

□ □ 0 



o 
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□ 1*1 



o <* wwi 



V. (. About how miny dm filmdi doit your Child D Non# Q 1 

(Do not Inctudi bfothifi I ilitirt) 

I. About ha« m.n, ttm.a . w..k doa. you. «MW <• wl,h •"» w *"<* •«»•'<• •• Wj|*' Y <W " t □ J 
(Do not KkMi broth,,, 1 .1.1...) □ la..th.o< Ul»> U 

VL Compirod lo olhin of hlifeir 191. how will don you* child: 

Wofio About Avirigo llttof 

□ 0 □ 

□ □ o 

□ DO 

□ □ □ 



vti. 



Of more 



0. Cll l , 0«g with ht|/h|f b<0th|fl 4 
b. Col ilong with olhOf kidll 
C. 6oh*vo with hivfhn pj/mli? 

4, ploy ond work by himsilUhorootf? 



Q H11 no Nolhon or 1I1I1 



1. V For ,g„ • and otd.r-p.rtarm.nc. In .e.d.mk awbJ.eU. M «Mld I. not batn 9 tau 9 hU *»•••. «hr. ra..an 



0. flooding, English, or languogo Aj 1 i 
b. Hliiory ot Sociot Stvdlii 
0. Arithmetic or Moth 
d. Science 



fining Bilow ivingo Avifige Abcti ivingo 

□ □ □ □ 

□ □ □ o 

□ □ □ □ 

□ □ □ □ 



Othor ecaderrdc pi p 

» Ww ot, a, — w w 


□ 

□ 


ample: computer p p 

COufftOO IfifikM f. — U CJ 


□ 

□ 


Unouago. boot- n n 

not*. Do not Wv L - 1 LJ 

ChKJo gym, $hop. 
ddvo<*i ed„ otc 


□ 

□ 


1. to your chHd In i ipeclit clou or ipoclol ochoott 0 No 


□ Yu - whof kind of elm o 4 school? 


1 Hoo your child fipoilod o gndot ® 


O Yii-gndt ond ronon 


4. Haa your ehlld bad any icad.mte ar othar yrobl.m. In tehoolt 0 No 

When did thuo probtomi ilirtt 

Hove thoio probtimi indodt 0 No O Yii-whiol 


O Y*. -pftata datcrtbo 


Dooi yo«f Child hofo ony lllnoio, phyitcol dliibfllly, k minfi! hindkip? 0 No 


O Yii-plioio doicffbo 



What concoma you moat about your chltdl 



ftoaao daacrlbo ff» baa I tWnf. about your «M* 



O 
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fcelow 1$ a H»t of Items that describe children. For each Item that describes your child now or within tha past 6 monihs 
cfrcta tha 2 if tha ttam is vary trua or oftan trua of your child. Clrcla tha 1 if tha itam it somawhat or sometimes trua 
child. If tha Ham Is not trua of your child, clrcla tha 0. P leasa answar all Hams as wall as you can, avan If soma do « 
to apply io your child. 

0 * Not Trua (as far as you know) 1 » Somawhat or Somatlmaa Trua 2 « Vary Trua or Ofta 



0 

0 

0 

0 

0 

0 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 



2 

2 

2 

2 

2 

2 



B 1. 

2 



"3. 

4. 



“5. 



Acts too young for his/h ar sga 
Allergy (describe): 



“9. 



Arguas a lot 
Asthma 

Behaves Ilka opposita sax 
Bowel movements ou!$ ; de toilet 

Bragging, boasting 
Can't concentrate, can’t pay attention for long 

Can’t get his/her mind off certain thoughts; 
obsessions (describe): 



0 12 ^0. Can’t sit still, restless, or hyperactive 



1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 



2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 



^11. Clings to adults or too dependent 
12. Complains of loneliness 

1 13. Confused or seems to be in a fog 
14. Cries a lot 

^15. Cruel to animals 
16. Cruelty, bullying, or meanness to others 

a 

^ 17. Day-dreams or gets lost In his/her thoughts 
16. Deliberately harms self or attempts suicide 

a 

^ 19. Demands a lot of attention 
20. Destroys his/her own things 

*21. Destroys things belonging to his/her family 
or other children 
22. Disobedient at home 

*23. Disobedient at school 

24. Doesn’t eat wall 

*25. Doesn’t gat along with other children 

* 26. Doesn’t seem to feel guilty after misbehaving 

*27. Easily Jealous 

26. Eats or drinks things that are not food- 
donl Include sweats (describe): 



0 1 



29. 



Fears certain animals, situations, or places, 
other than school (describe): 



30. Fears going to school 



0 

0 

0 

0 

0 

0 

0 

0 



0 

0 

0 

0 

0 

0 

0 

0 



0 

0 

0 

0 

0 

0 

0 

e 



*31. 



2 *32. 
2 a 33. 
a 

2 1 34. 
2 35. 



Fears he/she might think or do some 
bad 

Feels he/sha has to be perfect 
Feats or complains that no one love) 

Feels others are out to gat hinVher 
Feels worthless or inferior 



2 *36. Get* hurt • lot, eccldent-prone 
2 *37 



*36. 

*39. 



2 *40. 



0 1 2 "41. 



Gets In many fights 
Get* teased a lot 

Hangs around with children who get 
trouble 

Hears sounds or votcas that aren’t i 
(describe): 



42 

*43. 



2 *44. 
2 *45. 

2 *46. 



Impulsive or acts without thinking 

Likas to be alone 
lying or cheating 

Bites fingernails 

Nervous, highstrung, or tense 

Nervous movements Of twitching (<jei 



0 1 2 47. Nightmares 



2 *46. 
2 49. 

2 *50. 
2 *51. 

2*52 
2 53. 

2 *54. 
2 *55. 

56. 

2 

2 

2 

2 

2 
2 
2 
2 



Not liked by other children 
Constipated, doesn’t move bowel* 

Too fearful or anxloua 
Feel* dizzy 

Feel* too guilty 
Ovareatlng 

Overtired 

Overweight 

Physical problema without known m< 

cause 

'a. Ache* or paint 

l b. Headaches 

'c. Nausea, fetls tick 

1 d. Problems with syss (describe): 



«e 

•l 

a 



Rathe* or other akin problems 
Stomachaches or cramp* 

0. Vomiting, throwing up 
Ik Other (dsecrlbe): 



k'6 



BEST COPY AVAILABLE 



0- Not True far at you **ow) 



0 

0 



0 

0 

0 

0 

0 

0 

0 

0 



1 

1 

1 

1 

1 

1 

1 

1 



“* t JT. Physically attack! people ^ 

2 • se. Picks note, skin. Of olhsr part* of t>o<3y 

(doscrlbo): „ 



1 * Somewhat or Sometime* True 
0 " 



2 

2 

2 

2 

2 

2 

2 

2 



59. 

60. 

*6t. 

*62. 



Plays with own sex parts In public 
Plays with own sex parts too much 

Poor school work 

Poorly coordinated or clumsy 



* 63. Prefers being with older kids 

* 64. Prefers being with younger kids 



65. 

{ 66 . 



Refuses to talk 

Repeals certJn acts over and over, 
compulsions (describe): — 



0 1 2 67. Runs away from home 

0 12* 66 . Screams a lot 

0 1 2 * 69. Secretive* keeps things to self 

0 1 2 * 70. Sees things that aren't there (describe) 






2- Very True or Often True 



82 



Strange behavior (daacrlbe):. 



2 *65. 



Sirange Ideas (dascrlbe): 



0 1 2 * 66 Stubborn, sullen, or Irritable 



67, 

66 . 

*89. 

*90. 

* 01 . 

92. 



Sudden chenges In mood Of Uellng* 
Sulk* • lot 

Suspicious 

Sweerlng Of obscene Itngusgs 

Talks tt>oul killing sell 

Talks « walks In sleep (describe): _ 



0 1 2 * 71. Sell-conscious of eeslly embarrassed 

0 1 2 72. Sets liras 

73. Sexual problems (describe): 



2 4 93. Talks too much 
2 * 94. Teases a lot 

2 a 95 , Temper tantrums or hot temper 
2 A 96. Thinks about sex loo much 

2 * 97. Threatens people 
2 98. Thumb-sucking 

2 * 99. Too concerned wllh neatness or cltanllnes 
2 100. Trouble sleeping (describe): 



Truancy, skips school 

Underacllve, slow moving, or lacks energy 

Unhappy, sad. Of depressed 
Unusually loud 

Uses alcohol or drugs tor nonmedical 
purposes (describe):. 



O 
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Withdrawn, doesnT get Involved wllh others 
Worries 
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PROGRAM OUTCOME MEASURES 
Instructions 

As part of a Master’s level practicum project, you have been asked to provide information that 
will be used to develop and upgrade outcome evaluation practices in the short term program 
Please read each each section carefully, answer the corresponding questions by circling the 
appropriate response, and comment briefly in the spaces provided. Individual responses will 
remain confidential. Thank you for your cooperation. 
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PROGRAM OUTCOME MEASURES 

The Achenbach's Child Behavior Checklist (CBCL) and page 1 of the Parent Child 
Behavior Checklist (PCBQ) are designed to provide program staff with feedback on the 
treatment of troubled youth, that furthers the development of program research. At intake, 
specialists ask their clients to provide written consent to participate in program research. If 
consent is obtained, one or both parents complete these questionnaires within 2 weeks of 
opening, at case closure, and at 4 months following case closure. 

Part A 



( 1 ) How useful are the 2 existing instruments in measuring changes that take place within 
program families? 

Extremely Useful Useful Not Useful Not At All Useful 

Comments_ 

(2) How consistent is the questionnaire data with your own observations regarding family 
situations and progress toward goals? 

Highly Consistent Consistent Inconsistent Highly Inconsistent 

Comments — 



(3) From what percentage of your case families do you request consent to implement these 
measures? 

% 



Comments 



(4) How often do you issue these measures within 2 weeks of case opening with consenting 
families? 

Always Frequently Seldom Never 

Comments 



(5) How often do you issue these measures at case closure with consenting families? 

Always Frequently Seldom Never 

Comments 



(6) How often do you issue these measures at 4-month follow up with consenting families? 

Always Frequently Seldom Never 

Comments 
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(6) How often do you issue these measures at 4-month follow up with consenting families ’ 
Always Frequently Seldom Never 

Comments : — 



(7) How often do you use the questionnaire results to plan further clinical decisions 0 

Always Frequently Seldom Never 

C om ments 

(8) How often do you share the questionnaire results with other professionals (such as during 
case conferences, and in termination reports)? 

Always Frequently Seldom Never 

Comments 



(9) What percentage of your clients who are asked agree to participate in program research ? 

©/ 

/o 



Comments 



(10) How would you rate your own skills in statistically computing the questionnaire data 0 

Highly Skilled Skilled Unskilled Highly Unskilled 

Comments 



(11) How would you rate your ability to clinically interpret the questionnaires? 

Highly Capable Capable Incapable Extremely Incapable 

Comments . 

(12) How would you rate the adequacy of pre-service and in-service training in preparing you 
for the administration of the questionnaires and the interpretation of questionnaire data? 

Extremely Adequate Adequate Inadequate Extremely Inadequate 

Comments 



(13) How would you rate the questionnaires on their benefit to the program? 

Extremely Beneficial Beneficial Not beneficial Not At All Beneficial 

Comments 

(14) How difficult is it for you to find time to complete the questionnaires given your work 
demands? 

Extremely Difficult Difficult Easy Extremely Easy 

Comments 
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(15) In your opinion, to what degree are the benefits of research worthy of your time and 
energy 9 

Highly Worthy Worthy Unworthy Highly Unworthy 

Comments 

(16) When families decline participation in research, rate the importance of reasons 

(a) Thev do not understand what their commitment will involve. 

Extremely Important Important Unimportant Extremely Unimportant 

Comments 



(b) They feel it offers no immediate benefits to their family. 

Extremelv Important Important Unimportant Extremely Unimportant 

Comments 

(c) They are not satisfied with the ethical and confidentiality assurances program staff have 
made. 

Extremely Important Important Unimportant Extremely Unimportant 

Comments 



( 1 7) What percentage of the time do you find it difficult to contact families at 4-morith 
follow-up periods to issue the questionnaires 9 

% 



Comments 



Child placement is considered a primary outcome variable that provides program staff, 
professionals, and contract managers with feedback on program effectiveness. At case closure 
and thereafter over a 1-year period families are contacted quarterly by available staff and are 
asked to describe child placement during each month of the reporting period. 

Part B 

(18)1 low often do you collect child placement data during 1 -year follow-up intervals 9 

Always Frequently Seldom Never 

Comments — 
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(19) How consistent is the collection of child placement data during quarterly follow-up 
periods' 7 

Extremely Consistent Consistent Inconsistent Extremely Inconsistent 

Comments 

(20) How often do vou receive feedback on child placement data gathered by program staff 7 

Always Frequently Seldom Never 

Comments 

(21 ) How appropriate is the current outcome data package in providing sufficient family 
information to target groups (such as program staff and contract managers)' 7 

Extremely Appropriate Appropriate Inappropriate Extremely Inappropriate 

Comments 

(22) How appropriate is the existing child placement data as a single indicator of case outcome" 7 

Extremely Appropriate Appropriate Inappropriate Extremely Inappropriate 

Comments 

(23) How pressured do you feel to ensure in-home pi?, ,'menf 7 

Extremely Pressured Pressured Wot Prefaced Not At All Pressured 

Comments 
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in the following section please comment in more detail, or add any information not covered 
prev iously in the body of the questionnaire. 



Part C 

( 1 ) Comments related to the outcome measures used by the program: • 



2) Comments related to the consistent application of the outcome measures: 



(3) Comments related to the adequate use of data provided by the outcome measures: 




n 
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NOVA UNIVERSITY 

ABRAHAM S. FISCHLER CENTER FOR THE 
ADVANCEMENT OF EDUCATION 
3301 College Avenue 
Fort Lauderdale, Florida 33314 



VERIFICATION OF PRACTICUM PROJECT STATEMENT 
STATEMENT H - To be attached to the Practicum Report . 



by the student in a responsible, professional and competent manner. 

Practicum Title im&lLLRZJz EaR d. 



EnmiL±L /to R'Ujjui FirsinoN 



Verifier’s Position FqEJBZIL amLJjQiMBlh. THRU-LAE^ 

Verifier’s Relationship to Student EoRmsJl £m£LQ±±lR 

Name of Verifier’s Work Setting /- V/ cr>N\ u i Tin s L -T‘p. 



Telephone Number QlO, j) JJ2 LlQ&IsJL / tuLZuix. kAj 



I verify that the below named student did conduct the practicum project described in the 
submitted Practicum Report and I attest to the fact that this practicum project was carried out 



Student’s Name 6v?//. 



Cohort Sf- 



Verifier’s Name k@B±M. (kJiS. L: C Ehdl 



Address ^/9/cL ^a/ P/. f)C t 



CollajI y 



~ <3 M 



City 



State 



Zip 



Verifier’s Signature 




Date Jp^<- J, Ct % S 
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Family Assessment Device (FAD) 
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Enclosed please find the FAD packet that you ordered. You have permission 
to duplicate the copyrighted Family Assessment Device, the manual scoring 
sheet and instructions, and the Family Information Form. We may contact you 
in the future to receive your feedback on the instrument. 

Thank you for your interest and good luck-in-ycur future project. 

Sincerely, 

/* - » : ' 

if? 

''l van U. Miller, Ph . D. 

Director 

Brown University Family 
Research Program 
Butler Hospital 
345 Blackstone Blvd. 

Providence, R. I. 02906 



IWM/ 

Enclosure 
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FAMILY 

ASSESSMENT 

OSVICS 



Version 3 

Nathan B. Epstein, M.D. 

Lawrence M. Baldwin, Ph.D. 

Duane S. Bishop, M.D. 

The Brown University/Butler Hospital Family Research Program 

Butler Hospital 
345 Blackstone Boulevard 
Providence, Rhode Island 02906 



Date of Administration: 

Family Role: 

Identification Number or Family Name:- 
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FAMILY ASSESSMENT DEVICE 
Brown/ Bud«r c ttni\v R«*««reh Program 



ID 



P«S« l 



1. Planning family activities is difficult because we misunderstand each other. 

SA A D SD 

2. We resolve most everyday problems around the house. 

SA A D ■ SD 

3. Whc" someone is upset the others know why. 

SA A D SD 

4. When you ask someone to do something, you have to check that they did it 

SA A D SD 

5. If someone is in trouble, the others become too involved. 

SA A D SD 

6. In times of crisis we can turn to each other for support 

SA A D SD 

7. We don't know what to do when an emergency comes up. 

SA A D SD 

8. We sometimes run out of things that we need. 

SA A D SD 

9. We are reluctant to show our affection for each other. 

SA A D SD 

10. We make sure members meet their family responsibilities. 

SA A D SD 

1 1. We cannot talk to each other about the sadness we feel 

SA A D SD 

12. We usually act on our decisions regarding problems. 

SA A D SD 

O 
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ID 

FAMILY ASSESSMENT DEVICE 
Brown/ Sutler Ftmily Research Program 

13. You only get the interest of others when something is important to them. 

SA A D SD 

14. You can’t tell how a person is feeling from what they are saying. 

SA A D SD 

15. Family tasks don't get spread around enough. 

SA A D SD 

16. Individuals are accepted for what they are. 

SA A D SD 

17. You can easily get away with breaking the rules. 

SA A D SD 

18. People come right out and say things instead of hinting at them. 

SA A D SD 

19. Some of us just don’t respond emotionally. 

SA A D SD 

20. We know what to do in an emergency. 

SA A D SD 

21. We avoid discussing our fears and concerns. 

SA A D SD 

22. h is difficult to talk to each other about tender feelings. 

SA A D SD 

23. We have trouble meeting our bills. 

SA A D SD 

24. After our family tiles to solve a problem, we usually discuss whether It worked 

SA A D SD 



ID 

FAMILY ASSESSMENT DEVICE 
Broun/Sutfcr Family Rcaaarch Prosram 

25. We are too self-centered. 

SA A D SD 

26. We can express feelings to each other. 

SA A D SD 

27. We have no dear expectations about toilet habits. 

SA A D SD 

2S. We do not show our love for each other. 

SA A D SD 

29. We talk to people directly rather than through go-betweens. 

SA A D SD 

30. Each of us has particular duties and responsibilities. 

SA A D SD 

31. There are lots of bad feelings in the family. 

SA A D SD 

* 

32. We have rules about hitting people. 

SA A D SD 

33. We get involved with each other only when something interests us. 

SA A D SD 

34. There’s little time to explore personal interests. 

SA A D SD 

35. Wc often don’t say what we mean. 

SA A D SD 

36. • We feel accepted for what wc are. 

SA A D SD 

o 
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ID 



FAMILY ASSESSMENT DEVICE 
Brown/ Bulla* F«m<ly ftu««rch ProjTim 



37. We show interest in each other when we can get something out of it personally. 

SA A D ■ SD 

38. We resolve most emotional upsets that come up. 

SA A - D SD 

39. Tenderness takes second place to other things in our family. 

SA A D — SD 



40. We discuss who is to do household jobs. 
SA A D _ 



SD 



41. Making decisions is a problem for our family. 

SA A D SD — 

42. Our family shows interest in each other only when they can get something out of it 

SA A D SD — 



43. We are frank with each other. 
SA A 



44. We don't hold to any rules or standards. 
SA A D - 



SD 



SD 



45. If people are asked to do something, they need reminding. 

SA A D SD 

46. We are able to make decisions about how to solve problems. 

SA A 0 SO 

47. If the tules are broken, we don’t know what to et^ect 

SA _ A D SD 



46. Anything goes in our family. 

SA A - 



wJ 



O 
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ID 

FAMILY ASSESSMENT DEVICE 
8 rowny Sutler Pimtly ftuurch Pro^nrrt 

49. We express tenderness. 

SA A D SD 

50. We confront problems involving feelings. 

SA A D SD 

51. We don't get along wei! together. 

SA A D SD 

52. We don't talk to each other when we are angry. 

SA A D SD 

53. We are generally dissatisfied with the family duties assigned to us. 

SA A D SD 

54. Even though we mean well, we intrude too much into each others lives. 

SA A D SD 

55. There are rules about dangerous situations. 

SA A D SD 

56. We confide in each other. 

SA A D SD 

57. We ay openly. 

SA A D SD 

58. We don't have reasonable transport 

SA A D SD 

59. When we don't like what someone has done, we tell them. 

SA A D SD 

60: We try to think of different ways to solve problems. 

SA A D SD 

O 
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Personal Network Matrix (PNM) 
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Personal Network Matrix 
(Version 2) 

Carol M. Trivolt* 4 Cart J. ftwet 
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adit , 



0«t* 



piit questionnaire ask* about people and groups that may provide you hatp and assistance. Th* seat* is divided into thr** parts, 
pnasa raad th# instructions that qo with asch part bator* completing aach saction of th* questionnaire. 



often hav* oonUct with taca to fae*. in a group, or by 



gstad balow arc different individuals and groups that paopl* 
ptaasa Indical# lor each source listed how often you hav* boon in contact with aach parson or group during th* past 



How frequently hav* you had contact with aach 
pi the following during th* past month; 



14 . 

14 . 

ao. 

21 . 

XL 

24. 



Spous* or Partner 

My Children 

My Parents 

Spousa or Partner’s Parents 

My Sister/Brother 

My Spouse or Partner’s Sistar/Brothar 

Other Relatives 

Friends 

Neighbors 

Church Members 

MMster, Priest, or Rabbi 

Co-workers 

BebySitter. 

OayCsre or School 

Pifvsta Therapist for Child 

14. CNM/Family Doctors 

17 . Cady Childhood Intervention Program 1 

Hoapttal/Spedal OMca 

Heafth Department 

Soda! Service Department 

Other Age (Scies 



1 . 

2 . 

4 . 

4. 

5 . 

4. 

7 . 

4. 

4 . 

10 . 

11 . 

12 . 

14 . 

14. 

15. 





Not 


Once 


At Least 


At least 


Almost 


at 


or 


10 


20 


Every* 


AJ 1 


Twice 


Timas 


Timas 


Dey 




2 


9 


4 


S 




2 


9 


4 


S 


. .1 


2 


9 


4 


s 


. .1 


2 


9 


4 


5 



9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

4 

4 



ephon*. 

month. 
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h«lp or auiilanc*. he or the generally eon depend upon certain pereone or group* more then other*, liti 
below ore dlfferem IndMduele. group*, end agencies thet you might ask lor help or auiitane*. For eeeh *ouree listed, pie*** indie* 
«o whet extent you could dep*nd upon eeeh person or group If you needed any type o« help. 



To whet extend een you depend on eny of the 
following lor help or eselitenee when you need K: 



Spouse or Pertner 

My Children 

My Parent* 

Spouse or Pertner 1 * Parent*. 

My $i*ter/Brother 

My Spouse or Partner** Sister/Brother 

Other Relative* 

Friend* 

Neighbor* 

Church Member* 

Minister. Priest, or Rabbi 

Co wor k er* 

Baby Sitter 

Oey Car* or School 

Private Therapist lor Child 

Child/Famlty Ooctor* 

Early Childhood Intervention Program 

Hoeprtal/Spedal Olnic* 

HeaMh Department 

Social Servie* Department 

Other Agenda* 



Not 

at Some* 

M time* Occasionally 


Moil 
of the 
Time 


AD 

of the 
Tim* 


. 1 


9 


4 


5 


. 1 


9 


4 


5 


. 1 


9 


4 


5 


. t 


9 


4 


5 


. 1 


9 


4 


5 


. 1 


9 


4 


9 


. t 


9 


4 


9 


.1 


9 


4 


9 


.1 


9 


4 


9 


, .1 


9 


4 


9 


. .t 


9 


4 


9 


. .1 


9 


4 


9 


. .1 


9 


4 


9 


. .1 


9 


4 


9 


. .1 


9 


4 


9 


. .1 


9 


4 


9 


. .1 


9 


4 


9 


. .1 


9 


4 


9 


. .1 


9 


4 


9 


. .1 


9 


4 


9 


. .1 


9 


4 


9 


. . 1 


9 


4 


9 


. .1 


9 


4 


9 



Source: CJ. OunM. C M. Trtvett*. and A.0. 0**1 («••). FnaWng and empowering kmm$: Principal and puidefln** fort***** 
Cambridge. M* Brookfine Book*. May be reproduced. 




104 



RC3T COPY AVAILABLE 



APPENDIX E 

Family Goal Recording (FGR), Reproduced 
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FAHliY GOAL RECORDING (FGR): SUBSYSTEM COAL AREAS 



r. 
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APPENDIX F 

Post-Test Revisions to Specialist Questionnaire 
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PROGRAM OUTCOME MEASURES 



The Family Assessment Device (FAD), Personal Network Matrix (PNM), and Family Goal 
Recording (FGR) outcome measures are designed to provide program staff, clients, and target 
groups with consistent, relevant feedback that can assist with the assessment and treatment of 
children and families and determine client progress toward goals. At intake, clients must 
provide written consent to participate in counselling. Their commitment includes the 
completion of these evaluation instruments within 2 weeks of case opening and at case closure, 
and the completion of the FAD and PNM scales at 3 months following closure. Clients may also 
consent to the use of this data in program research. 

Part A 



( 1 ) How useful are the 3 existing instruments in measuring changes that take place within 
program families? 

Extremely Useful Useful Not Useful Not At All Useful 
Comments 



(2) How consistent is the questionnaire data with your own observations regarding family 
situations and progress toward goals? 

Highly Consistent Consistent Inconsistent Highly Inconsistent 

Comments 



(3) From what percentage of your case families do you request consent to implement these 
measures? 



Comments 



% 



(4) How often do you issue these measures within 2 weeks of case opening with consenting 
families? 

Always Frequently Seldom Never 

Comments 



(5) How often do you issue these measures at case closure with consenting families? 

Always Frequently Seldom Never 

Comments 




ill 
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Consent to Participate in Counselling Form, Revised 
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WILLIAM ROPER HULL CHILD AND FAMILY SERVICES 



108 



* . 



FAMILY INITIATIVES 

2266, Woodpark Avenue, S.W. Calgary, Alberta, T2W 2Z8 

CLIENT AUTHORIZATION FOR COUNSELLING 

The parent or legal guardian must sign this authorization for counselling form. 

1. Your sifftature authorizes your minor child(ren) to participate in the Family Initiatives Program. 

2 Your signature also authorizes the Family Initiatives' staff to work with other household members during 

the period of Family Initiatives' counselling and follow-up. 

X Your signature further authorizes Family Initiatives' staff to obtain relevant family information using 
questionnaires, interviews, and other documents, to assist you and them in meeting intervention goals. 



Name(s) of children: 

(Please print) 



Parent/Legal Guardian: 

(Please print) 



EMERGENCY CONTACT (optional) 

If Family Initiatives is unable to contact you in the event of an emergency, who may we 
contact? 

NAME: PHONE: 

(Please print) 



I have read and understand the above. 

SIGNATURE OF PARENT/LEGAL GUARDIAN: 

WITNESS: DATE: 

Calgary, Alberta, (403) 251-5770 

pd\afss\ctkaUtiirm 
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APPENDIX H 



Termination Format Letter, Revised Page 



The progress towards these goals was as follows: 

1 . 
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2 . 



3 . 



Points to consider when writing about goals: 

- Describe implementation of treatment 

• Chart progress, change, strengths 

- Realistically review ongoing problem areas 

- Use specific language describing behavioral change, impact on family dynamics, 
counselling outcomes, etc. 

• Link positive outcomes to intervention plan 

Include relevant statistical or clinical data to support your findings 
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Client Discharge Form, Revised Page 
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Code 


Service Description 


TT1 

Check One 


1 


Home of Parents 




2 


Relatives/Friends 




3 


Adoption 




4 


Home with Counselling 




5 


In-Home Support 




6 


Foster Care 




7 


Spec. Foster Care 




8 


Receiving Home 




9 


Group Home 




10 


Residential • Open 




11 


Residential • Closed 




12 


Remand in Custody 




13 


Supported Independent Living 




14 


Living Independently 




15 


Hospital (Physical) 




16 


Hospital (Psych) 




17 


Institutional 




18 


Runaway/AWOL 




19 


Other Parent 




20 


Emergency Shelter 




21 


Cedarbrae Teaching Home 




22 


Interdependent Living Services 




23 


Family Initiatives 




24 


Rsdissoo Receiving Home 




25 


First Choice Assessment 




26 


First Choice Parenting 
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Discharged According to Plan: Yes No If No: Projected Long Range Goal: 

Ni t"*r la Cm Bwwij W»ni *o P— By. Prim* (Psmittarioa) . 



How successful do you predict this child wQl be ia the figure? 



How favourable are the cooditious under which the youth is leaving the program? 

l.Vtry r*on bis. IFWontbte, INtrtStr FiwnbU sot UsfawnMt, S.UsfawsMe, 5.Veiy Usfawsbte, ». Usfcws. 

Comments Regarding Discharge Planning: 
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Outcome Evaluation Forms: Child Placement and Family Functioning 




O 

ERLC 



1 IS 



FAMILY INITIATIVES 



114 



OUTCOME EVALUATION DATA - CHILD PLACEMENT 
A. Reporting Period: 



Month of Follow-Up 





1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Child Sample Size 


























(n) 


























Child Placement 


























• Least Restrictive Environment 
(%) 


























•Restrictive Environment (%) 

•Permanent 

•Temporary 



























B. Reporting Period: 



Month of Follow-Up 





1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Child Sample Size 


























0>) 


























Child Placement 






















- 




• Least Restrictive Environment 
(%) 


























•Restrictive Environment (%) 

•Permanent 

•Temporary 



























• Least restrictive placement = family, relatives, friends 

• Restrictive placement * province-funded in care 

• Long-range plan (projected): permanently in care/temporarily in care (goal - reunification) 

/ 
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FAMILY INITIATIVES 

OUTCOME EVALUATION DATA - FAMILY FUNCTIONING 



Reporting Period: 



Reporting Period: 



• FAMILY FUNCTIONING 
(FAD) 


DIMENSIONS 

(7) 


OPENING 

(n« ) 


CLOSING 
(n « ) 


FOLLOW-UP 
(»* ) 


Problem Solving 








Communication 








Roles 








Affective 

Responsiveness 








Affective 

Involvement 








Behavior 

Control 








General 

Functioning 









• FAMILY FUNCTIONING 
(FAD) 


OPENING 

(n« ) 


CLOSING 
(n« ) 


FOLLOW-UP 
(n« ) 













































* Family Assessment Device (FAD) 
Healthy 1 — / — 4 Unhealthy 
2 cutoff 
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